Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

storage: docs & rewrite streams. #114

Merged
merged 1 commit into from
Aug 18, 2014
Merged

Conversation

stephenplusplus
Copy link
Contributor

ReadableStream and BufferStream implement stream.Readable, and I created a WritableStream, which works similar to Readable, but obviously in reverse :)

This also includes elongated documentation.

*/
function ReadStream(bucket, name) {
events.EventEmitter.call(this);
stream.Readable.call(this);

This comment was marked as spam.

This comment was marked as spam.

@stephenplusplus
Copy link
Contributor Author

Will revert to docs only. 📝 (Thanks for reviewing!)

@rakyll
Copy link
Contributor

rakyll commented Aug 16, 2014

I have limited experience with Streaming API while implementing this client. I initially wanted to provide Readable and Writable (maybe draft code was still available in the history), but ended up having buffering the entire file contents in memory due to the cases I've experienced above.

If we can fix these problems and still keep inheriting Readable and Writable, I'm all for that.

@stephenplusplus
Copy link
Contributor Author

If I understand right, the only reason we can't implement streams is due to the [possibly] async call to get a token. Is there another way we can get a token before allowing streams to be opened? Either with events:

function runMyFunctionThatOpensAStream() {
  fs.createReadStream('hugefile').pipe(bucket.createWriteStream('file'));
}

bucket.on('authenticated', runMyFunctionThatOpensAStream);

or callbacks:

bucket.stream(runMyFunctionThatOpensAStream);

? Can you see either of those working/being usable?

@stephenplusplus
Copy link
Contributor Author

I've been reading more on streams, and I think we may have nothing to worry about.

readable.pipe
This method pulls all the data out of a readable stream, and writes it to the supplied destination, automatically managing the flow so that the destination is not overwhelmed by a fast readable stream.

To indicate that we're "overwhelmed," I believe all have to do is not execute the callback provided to OurWritableStream._write.

I also discovered highWaterMark:

The maximum number of bytes to store in the internal buffer before ceasing to read from the underlying resource. Default=16kb

The default value is 64kb for fs.createReadStream.

This is telling me we are dealing with "pull streams" as opposed to "push". I'm still wrapping my head around the stream API, but these discoveries left me feeling more confident.

(This PR would still require re-doing to implement this stuff correctly.)

I opened a question on stackoverflow: http://stackoverflow.com/questions/25355788/does-a-readable-start-reading-before-a-writable-emits-writable

Stream article that finally helped me: http://howtonode.org/streams-explained 👍

@stephenplusplus
Copy link
Contributor Author

My Stream re-write is out. I'm optimistic that we can make them work, but, y'know, some date in the future.

* Makes a new request object from the provided
* arguments, and wraps the callback to intercept
* non-successful responses.
* Make a new request object from the provided arguments and wrap the callback

This comment was marked as spam.

rakyll pushed a commit that referenced this pull request Aug 18, 2014
storage: docs & rewrite streams.
@rakyll rakyll merged commit 362b6a8 into googleapis:master Aug 18, 2014
@rakyll
Copy link
Contributor

rakyll commented Aug 18, 2014

LGTM, merging.

I will come back to streaming hurdles soon.

@stephenplusplus stephenplusplus deleted the storage branch August 19, 2014 19:11
sofisl pushed a commit that referenced this pull request Oct 13, 2022
This PR was generated using Autosynth. 🌈

Synth log will be available here:
https://source.cloud.google.com/results/invocations/b742586e-df31-4aac-8092-78288e9ea8e7/targets

- [ ] To automatically regenerate this PR, check this box.

PiperOrigin-RevId: 325949033
Source-Link: googleapis/googleapis@94006b3
sofisl pushed a commit that referenced this pull request Oct 13, 2022
…114)

This PR was generated using Autosynth. 🌈

Synth log will be available here:
https://source.cloud.google.com/results/invocations/36077dab-6428-471f-9375-a14a1a227f45/targets

- [ ] To automatically regenerate this PR, check this box.

Source-Link: googleapis/synthtool@8cf6d28
sofisl pushed a commit that referenced this pull request Oct 13, 2022
[![WhiteSource Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com)

This PR contains the following updates:

| Package | Change | Age | Adoption | Passing | Confidence |
|---|---|---|---|---|---|
| [@types/node](https://togithub.com/DefinitelyTyped/DefinitelyTyped) | [`^12.12.34` -> `^14.0.0`](https://renovatebot.com/diffs/npm/@types%2fnode/12.20.13/14.17.0) | [![age](https://badges.renovateapi.com/packages/npm/@types%2fnode/14.17.0/age-slim)](https://docs.renovatebot.com/merge-confidence/) | [![adoption](https://badges.renovateapi.com/packages/npm/@types%2fnode/14.17.0/adoption-slim)](https://docs.renovatebot.com/merge-confidence/) | [![passing](https://badges.renovateapi.com/packages/npm/@types%2fnode/14.17.0/compatibility-slim/12.20.13)](https://docs.renovatebot.com/merge-confidence/) | [![confidence](https://badges.renovateapi.com/packages/npm/@types%2fnode/14.17.0/confidence-slim/12.20.13)](https://docs.renovatebot.com/merge-confidence/) |

---

### Configuration

📅 **Schedule**: "after 9am and before 3pm" (UTC).

🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied.

♻️ **Rebasing**: Whenever PR is behind base branch, or you tick the rebase/retry checkbox.

🔕 **Ignore**: Close this PR and you won't be reminded about this update again.

---

 - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box.

---

This PR has been generated by [WhiteSource Renovate](https://renovate.whitesourcesoftware.com). View repository job log [here](https://app.renovatebot.com/dashboard#github/googleapis/nodejs-memcache).
sofisl pushed a commit that referenced this pull request Oct 13, 2022
Co-authored-by: Takashi Matsuo <tmatsuo@google.com>
sofisl pushed a commit that referenced this pull request Nov 11, 2022
sofisl pushed a commit that referenced this pull request Nov 11, 2022
sofisl pushed a commit that referenced this pull request Nov 11, 2022
[![Mend Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com)

This PR contains the following updates:

| Package | Change | Age | Adoption | Passing | Confidence |
|---|---|---|---|---|---|
| [jsdoc-fresh](https://togithub.com/googleapis/jsdoc-fresh) | [`^1.0.2` -> `^2.0.0`](https://renovatebot.com/diffs/npm/jsdoc-fresh/1.1.1/2.0.0) | [![age](https://badges.renovateapi.com/packages/npm/jsdoc-fresh/2.0.0/age-slim)](https://docs.renovatebot.com/merge-confidence/) | [![adoption](https://badges.renovateapi.com/packages/npm/jsdoc-fresh/2.0.0/adoption-slim)](https://docs.renovatebot.com/merge-confidence/) | [![passing](https://badges.renovateapi.com/packages/npm/jsdoc-fresh/2.0.0/compatibility-slim/1.1.1)](https://docs.renovatebot.com/merge-confidence/) | [![confidence](https://badges.renovateapi.com/packages/npm/jsdoc-fresh/2.0.0/confidence-slim/1.1.1)](https://docs.renovatebot.com/merge-confidence/) |

---

### Release Notes

<details>
<summary>googleapis/jsdoc-fresh</summary>

### [`v2.0.0`](https://togithub.com/googleapis/jsdoc-fresh/blob/HEAD/CHANGELOG.md#&#8203;200-httpsgithubcomgoogleapisjsdoc-freshcomparev111v200-2022-05-18)

[Compare Source](https://togithub.com/googleapis/jsdoc-fresh/compare/v1.1.1...v2.0.0)

##### ⚠ BREAKING CHANGES

-   update library to use Node 12 ([#&#8203;108](https://togithub.com/googleapis/jsdoc-fresh/issues/108))

##### Build System

-   update library to use Node 12 ([#&#8203;108](https://togithub.com/googleapis/jsdoc-fresh/issues/108)) ([e61c223](https://togithub.com/googleapis/jsdoc-fresh/commit/e61c2238db8900e339e5fe7fb8aea09642290182))

##### [1.1.1](https://www.github.com/googleapis/jsdoc-fresh/compare/v1.1.0...v1.1.1) (2021-08-11)

##### Bug Fixes

-   **build:** migrate to using main branch ([#&#8203;83](https://www.togithub.com/googleapis/jsdoc-fresh/issues/83)) ([9474adb](https://www.github.com/googleapis/jsdoc-fresh/commit/9474adbf0d559d319ff207397ba2be6b557999ac))

</details>

---

### Configuration

📅 **Schedule**: Branch creation - "after 9am and before 3pm" (UTC), Automerge - At any time (no schedule defined).

🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied.

♻ **Rebasing**: Whenever PR is behind base branch, or you tick the rebase/retry checkbox.

🔕 **Ignore**: Close this PR and you won't be reminded about this update again.

---

 - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, click this checkbox.

---

This PR has been generated by [Mend Renovate](https://www.mend.io/free-developer-tools/renovate/). View repository job log [here](https://app.renovatebot.com/dashboard#github/googleapis/nodejs-api-gateway).
sofisl pushed a commit that referenced this pull request Nov 11, 2022
* test: use fully qualified request type name in tests

PiperOrigin-RevId: 475685359

Source-Link: googleapis/googleapis@7a12973

Source-Link: googleapis/googleapis-gen@370c729
Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiMzcwYzcyOWUyYmEwNjJhMTY3NDQ5YzI3ODgyYmE1ZjM3OWM1YzM0ZCJ9

* 🦉 Updates from OwlBot post-processor

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com>
sofisl pushed a commit that referenced this pull request Nov 11, 2022
samples: pull in latest typeless bot, clean up some comments

Source-Link: https://togithub.com/googleapis/synthtool/commit/0a68e568b6911b60bb6fd452eba4848b176031d8
Post-Processor: gcr.io/cloud-devrel-public-resources/owlbot-nodejs:latest@sha256:5b05f26103855c3a15433141389c478d1d3fe088fb5d4e3217c4793f6b3f245e
sofisl pushed a commit that referenced this pull request Nov 11, 2022
[![Mend Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com)

This PR contains the following updates:

| Package | Change | Age | Adoption | Passing | Confidence |
|---|---|---|---|---|---|
| [jsdoc-region-tag](https://togithub.com/googleapis/jsdoc-region-tag) | [`^1.0.6` -> `^2.0.0`](https://renovatebot.com/diffs/npm/jsdoc-region-tag/1.3.1/2.0.0) | [![age](https://badges.renovateapi.com/packages/npm/jsdoc-region-tag/2.0.0/age-slim)](https://docs.renovatebot.com/merge-confidence/) | [![adoption](https://badges.renovateapi.com/packages/npm/jsdoc-region-tag/2.0.0/adoption-slim)](https://docs.renovatebot.com/merge-confidence/) | [![passing](https://badges.renovateapi.com/packages/npm/jsdoc-region-tag/2.0.0/compatibility-slim/1.3.1)](https://docs.renovatebot.com/merge-confidence/) | [![confidence](https://badges.renovateapi.com/packages/npm/jsdoc-region-tag/2.0.0/confidence-slim/1.3.1)](https://docs.renovatebot.com/merge-confidence/) |

---

### Release Notes

<details>
<summary>googleapis/jsdoc-region-tag</summary>

### [`v2.0.0`](https://togithub.com/googleapis/jsdoc-region-tag/blob/HEAD/CHANGELOG.md#&#8203;200-httpsgithubcomgoogleapisjsdoc-region-tagcomparev131v200-2022-05-20)

[Compare Source](https://togithub.com/googleapis/jsdoc-region-tag/compare/v1.3.1...v2.0.0)

##### ⚠ BREAKING CHANGES

-   update library to use Node 12 ([#&#8203;107](https://togithub.com/googleapis/jsdoc-region-tag/issues/107))

##### Build System

-   update library to use Node 12 ([#&#8203;107](https://togithub.com/googleapis/jsdoc-region-tag/issues/107)) ([5b51796](https://togithub.com/googleapis/jsdoc-region-tag/commit/5b51796771984cf8b978990025f14faa03c19923))

##### [1.3.1](https://www.github.com/googleapis/jsdoc-region-tag/compare/v1.3.0...v1.3.1) (2021-08-11)

##### Bug Fixes

-   **build:** migrate to using main branch ([#&#8203;79](https://www.togithub.com/googleapis/jsdoc-region-tag/issues/79)) ([5050615](https://www.github.com/googleapis/jsdoc-region-tag/commit/50506150b7758592df5e389c6a5c3d82b3b20881))

</details>

---

### Configuration

📅 **Schedule**: Branch creation - "after 9am and before 3pm" (UTC), Automerge - At any time (no schedule defined).

🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied.

♻ **Rebasing**: Whenever PR is behind base branch, or you tick the rebase/retry checkbox.

🔕 **Ignore**: Close this PR and you won't be reminded about this update again.

---

 - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, click this checkbox.

---

This PR has been generated by [Mend Renovate](https://www.mend.io/free-developer-tools/renovate/). View repository job log [here](https://app.renovatebot.com/dashboard#github/googleapis/nodejs-org-policy).
sofisl pushed a commit that referenced this pull request Nov 16, 2022
…new_site_to_site_data_transfer_spokes field docs: Update comments to reflect that spokes can now be created with data transfer disabled (#114)

* feat: Add LocationMetadata message feat: Add RoutingVPC.required_for_new_site_to_site_data_transfer_spokes field docs: Update comments to reflect that spokes can now be created with data transfer disabled

PiperOrigin-RevId: 426937195

Source-Link: googleapis/googleapis@cb96586

Source-Link: googleapis/googleapis-gen@b535b72
Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiYjUzNWI3MjY4OTVkMmVjNTFlNDNiZTQ4ZWYyMmIwMWI2ZTFkNGQ0ZiJ9

* 🦉 Updates from OwlBot

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com>
sofisl pushed a commit that referenced this pull request Nov 16, 2022
[![WhiteSource Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com)

This PR contains the following updates:

| Package | Change | Age | Adoption | Passing | Confidence |
|---|---|---|---|---|---|
| [sinon](https://sinonjs.org/) ([source](https://togithub.com/sinonjs/sinon)) | [`^11.0.0` -> `^12.0.0`](https://renovatebot.com/diffs/npm/sinon/11.1.2/12.0.1) | [![age](https://badges.renovateapi.com/packages/npm/sinon/12.0.1/age-slim)](https://docs.renovatebot.com/merge-confidence/) | [![adoption](https://badges.renovateapi.com/packages/npm/sinon/12.0.1/adoption-slim)](https://docs.renovatebot.com/merge-confidence/) | [![passing](https://badges.renovateapi.com/packages/npm/sinon/12.0.1/compatibility-slim/11.1.2)](https://docs.renovatebot.com/merge-confidence/) | [![confidence](https://badges.renovateapi.com/packages/npm/sinon/12.0.1/confidence-slim/11.1.2)](https://docs.renovatebot.com/merge-confidence/) |

---

### Release Notes

<details>
<summary>sinonjs/sinon</summary>

### [`v12.0.1`](https://togithub.com/sinonjs/sinon/blob/master/CHANGES.md#&#8203;1201)

[Compare Source](https://togithub.com/sinonjs/sinon/compare/v12.0.0...v12.0.1)

-   [`3f598221`](https://togithub.com/sinonjs/sinon/commit/3f598221045904681f2b3b3ba1df617ed5e230e3)
    Fix issue with npm unlink for npm version > 6 (Carl-Erik Kopseng)
    > 'npm unlink' would implicitly unlink the current dir
    > until version 7, which requires an argument
-   [`51417a38`](https://togithub.com/sinonjs/sinon/commit/51417a38111eeeb7cd14338bfb762cc2df487e1b)
    Fix bundling of cjs module ([#&#8203;2412](https://togithub.com/sinonjs/sinon/issues/2412)) (Julian Grinblat)
    > -   Fix bundling of cjs module
    >
    > -   Run prettier

*Released by [Carl-Erik Kopseng](https://togithub.com/fatso83) on 2021-11-04.*

#### 12.0.0

### [`v12.0.0`](https://togithub.com/sinonjs/sinon/compare/v11.1.2...v12.0.0)

[Compare Source](https://togithub.com/sinonjs/sinon/compare/v11.1.2...v12.0.0)

</details>

---

### Configuration

📅 **Schedule**: "after 9am and before 3pm" (UTC).

🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied.

♻ **Rebasing**: Whenever PR is behind base branch, or you tick the rebase/retry checkbox.

🔕 **Ignore**: Close this PR and you won't be reminded about this update again.

---

 - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, click this checkbox.

---

This PR has been generated by [WhiteSource Renovate](https://renovate.whitesourcesoftware.com). View repository job log [here](https://app.renovatebot.com/dashboard#github/googleapis/nodejs-retail).
sofisl pushed a commit that referenced this pull request Nov 18, 2022
sofisl pushed a commit that referenced this pull request Jan 24, 2023
sofisl pushed a commit that referenced this pull request Jan 25, 2023
sofisl pushed a commit that referenced this pull request Sep 14, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants