Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: refactor to node streams #14

Merged
merged 3 commits into from
Nov 1, 2019

Conversation

ronag
Copy link
Contributor

@ronag ronag commented Oct 25, 2019

Refactors module into node streams. There are some rules that should be followed to maintain compatibility with node streams:

  • Don't override destroy()
  • Don't override open() (it's deprecated)

This fixes breakage in Node 13 nodejs/node#30110.

This is potentially breaking.

Fixes a bug where the capacitor could be destroyed before all readables have been destroyed.

Re-use same fd between write and read.

Slightly slower read since it doesn't re-use the pooling logic from ReadStream.

@mike-marcacci
Copy link
Owner

Hi @ronag,

Thanks so much for opening this and addressing the compatibility issues with node 13. I've skimmed your changes and am very interested in this approach. I will give this a deep dive tomorrow.

@@ -2,91 +2,62 @@ import crypto from "crypto";
import fs from "fs";
import os from "os";
import path from "path";
import { Readable, Writable } from "readable-stream";
Copy link
Contributor Author

@ronag ronag Oct 27, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

readable-stream is needed for Node 8 compat


this.bytesWritten = 0;
this.pos = 0;
this.closed = false;
Copy link
Contributor Author

@ronag ronag Oct 27, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

closed and bytesWritten are not really needed but I'm not sure what the expected public API here is and they remain to not unnecessarily further break compat. But I would recommend deprecation and removal.

if (!error) this.emit("write");
callback(error);
});
get finished() {
Copy link
Contributor Author

@ronag ronag Oct 27, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is also not really needed and stream in Node 13+ has a writableFinished property. Remains to not unnecessarily further break compat but I would recommend deprecation and removal. The readable side could just as easily access the internal property.

src/index.mjs Show resolved Hide resolved

this._readStreams = new Set();
this.error = null;
Copy link
Contributor Author

@ronag ronag Oct 27, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed error prop as it's a bit strange. Though I'm unsure how important it is for the public API and is a semver-major breaking change that needs to be considered.

this.once("open", this._destroy.bind(this, error, callback));
return;
}
if (this.destroyed) return;
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This won't be required once readable-stream is updated to Node 13 impl. However, before that Readable does not check for destruction before calling _read and this condition is therefore required.

this.error = error;
});

this.open();
}

get ended() {
Copy link
Contributor Author

@ronag ronag Oct 27, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is also not really needed and stream in Node 13+ has a readableEnded property. Remains to not unnecessarily further break compat but I would recommend deprecation and removal. The readable side could just as easily access the internal property.

Copy link
Owner

@mike-marcacci mike-marcacci left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is excellent; thanks for the self-review and explanations here. I have some other changes I'm going to add on top of this, but I think this will be far more portable than extending node's built-in file streams.

@mike-marcacci mike-marcacci merged commit cf6b9af into mike-marcacci:master Nov 1, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants