Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add documentation for Streams API #1576

Merged
merged 5 commits into from
May 13, 2024
Merged

Add documentation for Streams API #1576

merged 5 commits into from
May 13, 2024

Conversation

oleiade
Copy link
Member

@oleiade oleiade commented Apr 30, 2024

What?

This Pull Request adds documentation for the new k6/experimental/streams module.

Checklist

  • I have used a meaningful title for the PR.
  • I have described the changes I've made in the "What?" section above.
  • I have performed a self-review of my changes.
  • I have run the npm start command locally and verified that the changes look good.
  • I have made my changes in the docs/sources/next folder of the documentation.

@oleiade oleiade added the Area: OSS Content Improvements or additions to community/oss documentation label Apr 30, 2024
@oleiade oleiade requested review from a team, codebien and olegbespalov and removed request for a team April 30, 2024 12:49
@codebien codebien mentioned this pull request Apr 30, 2024
25 tasks
Copy link
Collaborator

@heitortsergent heitortsergent left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left a few comments but they're mostly small edits and changes to make it consistent with other parts of the docs. :) There is one empty example in the "releaseLock.md" file.

Overall it looks great, thanks for writing these docs! 🙇

},

// The pull function is called repeatedly to get data, while the
// internal high water mark is not reached.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is "high water mark" referring to the end of the file?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The high water mark is a streams-specific concept. It's representing the amount of chunks a stream can pull/contain before blocking.

For instance a stream with a high water mark of five, will holds at most 5 chunks in its internal queue. So if the client is really slow to pull data from the stream, we can control the total memory and CPU usage that the streams consume for instance.

@oleiade oleiade merged commit f43db08 into main May 13, 2024
5 checks passed
@oleiade oleiade deleted the document-streams-api branch May 13, 2024 10:18
Copy link
Contributor

@olegbespalov olegbespalov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was a bit late with the review, so LGTM, in general, and my comments could be applied optionally in other PRs.

One thing that I feel is important to mention is Streams API from another page, like https://grafana.com/docs/k6/latest/testing-guides/running-large-tests/ where we name options about improved memory consumption.


```javascript
import { CountQueuingStrategy, ReadableStream } from 'k6/experimental/streams';
import { setTimeout } from 'k6/timers';
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
import { setTimeout } from 'k6/timers';


Streaming involves breaking a resource that one wants to process or consume into smaller chunks, which can be processed or consumed incrementally. This is particularly useful when dealing with large files or data sources, as it allows you to work with the data in smaller, more manageable pieces.

With the Streams API support in k6, you can start processing raw data with Javascript bit by bit, as soon as it's available, without needing to generate a full in-memory representation of the data. This helps to reduce memory usage and improve performance, especially when working with large data sets.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
With the Streams API support in k6, you can start processing raw data with Javascript bit by bit, as soon as it's available, without needing to generate a full in-memory representation of the data. This helps to reduce memory usage and improve performance, especially when working with large data sets.
With the Streams API support in k6, you can start processing raw data with JavaScript bit by bit, as soon as it's available, without needing to generate a full in-memory representation of the data. This helps to reduce memory usage and improve performance, especially when working with large data sets.


```javascript
import { ReadableStream } from 'k6/experimental/streams';
import { setTimeout } from 'k6/timers';
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
import { setTimeout } from 'k6/timers';


```javascript
import { ReadableStream } from 'k6/experimental/streams';
import { setTimeout } from 'k6/timers';
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
import { setTimeout } from 'k6/timers';


```javascript
import { ReadableStream } from 'k6/experimental/streams';
import { setTimeout } from 'k6/timers';
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
import { setTimeout } from 'k6/timers';


```javascript
import { ReadableStream } from 'k6/experimental/streams';
import { setTimeout } from 'k6/timers';
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
import { setTimeout } from 'k6/timers';


```javascript
import { ReadableStream } from 'k6/experimental/streams';
import { setTimeout } from 'k6/timers';
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
import { setTimeout } from 'k6/timers';


```javascript
import { ReadableStream } from 'k6/experimental/streams';
import { setTimeout } from 'k6/timers';
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
import { setTimeout } from 'k6/timers';


## Returns

A promise which fullfils or rejects with a value depending on the state of the stream:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
A promise which fullfils or rejects with a value depending on the state of the stream:
A promise which fulfills or rejects with a value depending on the state of the stream:

@oleiade
Copy link
Member Author

oleiade commented May 14, 2024

I'll address your comments in a separate PR 🙏🏻 🙇🏻

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Area: OSS Content Improvements or additions to community/oss documentation
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants