-
Notifications
You must be signed in to change notification settings - Fork 28.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
fs: add FileHandle.prototype.readLines
#42590
Conversation
//cc @nodejs/fs @nodejs/readline |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think convenience methods like this one which isn't strictly necessary for the API to be fully functional should belong to userland modules like https://github.com/jprichardson/node-fs-extra.
@RaisinTen I disagree, I think it would be a shame to have to use a third party module for something like this. Other languages have this feature built-in:
Additionally, it's very little code to maintain on out end, and saves our users a good chunk of boilerplate. |
doc/api/fs.md
Outdated
console.log(line); | ||
} | ||
} finally { | ||
await file?.close(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Could this be handled automatically by the async iterator?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Oh it is indeed, autoClose
defaults to true
so that explicit close
call is superfluous.
I think python is the only language in this list that has an equivalent API for what we are doing in this PR.
Aren't both of these like our
But what's wrong with using an npm module? Those have loads of utilities already to reduce boilerplate code. |
I'm not saying using an npm package is wrong, and I don't think adding this utility would change anything to the download stats of e.g. |
To be clear, I'm totally OK to move this to the |
If we don't want to audit an external package, I think it's okay if we use the boilerplate code. I don't think the core repo is supposed to solve the use case of reducing boilerplate code.
I still don't think that would be a good idea because readline is supposed to be an interface - a wrapper around any sort of readable stream we want to read lines from. If we allow folks to use it in a way that's not like using an interface, I feel it kinda goes against the principles of that module. |
My use case is the following:
Is your response to this: "just use Python"? I don't really get how this util being part of core is detrimental to you or anyone (and if it's not detrimental, why do you block it?). |
Why don't you want to use npm? It would be really helpful to reduce a lot of boilerplate code throughout your application. Even if you are against using npm, you could create a utility function in your application and use that wherever you need to read lines from a file. That way, you could write the boilerplate code just once and forget it and use the more approachable utility function everywhere you need it - no need to learn anything by heart. Maybe you could put that into a private package repo if you don't want that utility to be there in your main application source?
That's a very valid solution.
It's not detrimental, it's very helpful. It just doesn't look like something that should belong to core because of the reasons I stated above.
If we absolutely do have to add this functionality to core, I would start off by adding it to readable streams, so that all sorts of readable streams can benefit from this (readline accepts a readable stream as the input object for that purpose). If it gets accepted there then I'm okay with adding it to fs if the createReadStream() part looks too much. |
Sorry, but what reasons? The only reason I'm reading is that you don't think it belongs in core. @RaisinTen Let's assume that:
And now with all that, I'm telling you: I still want this in core, I think it belongs there. I working with Node.js because I find it fun, and on Node.js core development because it let me overcome the frustrations I have with it sometimes. If you are not going to let me fix them when I found one, well that's not cool. Node.js is a community-owned project, refusing a feature on the basis that you don't like it is not acceptable IMHO, so if you agree that it's a useful feature and would not be detrimental as far as you know, then please remove your block.
TBF that's already a one-liner: import { createInterface as readLines } from 'node:readline'
for await(const line of readLines(myStream)) console.log(line) But that's not really my use-case anyway, mine is specifically about open file handles. |
This is my reasoning:
Well, there is a one-liner for this too for await(const line of readLines(openFileHandle.createReadStream())) console.log(line) |
I'll let others from @nodejs/fs and @nodejs/readline decide if we should add this
Actually, please disregard my second point. Could you please share what the API would look like if we expose this from readline? |
Thanks for clarifying, I didn't get that point, I think it's reasonable (although I don't agree 😅).
That's the gotcha: you need to provide for await(const line of readLines({ input: openFileHandle.createReadStream(), crlfDelay: Infinity })) console.log(line)
Well I suppose the easiest would be to add support for import { createInterface as readLines } from 'node:readline';
import { open } from 'node:fs/promises';
const fh = await open('path/to/file', 'r');
for await(const line of readLines(fh)) console.log(line); |
Does it sound reasonable to change the crlfDelay default to Infinity? Not sure if the one-liner would satisfy your needs then. |
@nodejs/fs @nodejs/readline can I get some more reviews please? |
For anyone else following along, the other half of the discussion regarding the inclusion of this functionality in core took place in #43102. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
+1 on having this method in fs
; I think that importance of builtin utility functions is underestimated: their absence might be an invisible obstacle for the whole language to become a convenient utility tool for daily routine.
It doesn't create inconsistency between low- and high-level API here, since we already have .readFile
.writeFile
anyway.
Well, the whole readline module is a builtin utility API. I don't see why we should create additional APIs on fs which internally just calls a single public API of readline given that it can be done very easily in userland, which is why we never got so many complaints about I feel that landing this change would have some consequences:
I think it's only fair to add a builtin utility function to core if it's tedious to implement in userland using public APIs of Node.js.
|
I agree that a utility to read a stream of lines from an arbitrary data source, whether it's a node stream or a web stream or a file handle, is desirable -- the lack thereof might be why I often switch to Python whenever I have to process line-based input data. The |
8afdcf4
to
ed06d25
Compare
ed06d25
to
1b9c406
Compare
@tniessen do you think it's ok to land this now that |
1b9c406
to
f85d840
Compare
PR-URL: #42590 Reviewed-By: LiviaMedeiros <livia@cirno.name>
Notable changes: * cli: * (SEMVER-MINOR) add `--watch` (Moshe Atlow) #44366 * fs: * (SEMVER-MINOR) add `FileHandle.prototype.readLines` (Antoine du Hamel) #42590 * http: * (SEMVER-MINOR) add writeEarlyHints function to ServerResponse (Wing) #44180 * http2: * (SEMVER-MINOR) make early hints generic (Yagiz Nizipli) #44820 * src: * (SEMVER-MINOR) add detailed embedder process initialization API (Anna Henningsen) #44121 * util: * (SEMVER-MINOR) add default value option to parsearg (Manuel Spigolon) #44631
Notable changes: * cli: * (SEMVER-MINOR) add `--watch` (Moshe Atlow) #44366 * fs: * (SEMVER-MINOR) add `FileHandle.prototype.readLines` (Antoine du Hamel) #42590 * http: * (SEMVER-MINOR) add writeEarlyHints function to ServerResponse (Wing) #44180 * http2: * (SEMVER-MINOR) make early hints generic (Yagiz Nizipli) #44820 * src: * (SEMVER-MINOR) add detailed embedder process initialization API (Anna Henningsen) #44121 * util: * (SEMVER-MINOR) add default value option to parsearg (Manuel Spigolon) #44631 PR-URL: #44968
Notable changes: * cli: * (SEMVER-MINOR) add `--watch` (Moshe Atlow) #44366 * fs: * (SEMVER-MINOR) add `FileHandle.prototype.readLines` (Antoine du Hamel) #42590 * http: * (SEMVER-MINOR) add writeEarlyHints function to ServerResponse (Wing) #44180 * http2: * (SEMVER-MINOR) make early hints generic (Yagiz Nizipli) #44820 * lib: * (SEMVER-MINOR) refactor transferable AbortSignal (flakey5) #44048 * src: * (SEMVER-MINOR) add detailed embedder process initialization API (Anna Henningsen) #44121 * util: * (SEMVER-MINOR) add default value option to parsearg (Manuel Spigolon) #44631 PR-URL: #44968
watch mode (experimental): Running in 'watch' mode using `node --watch` restarts the process when an imported file is changed. Contributed by Moshe Atlow in [#44366](#44366) Other notable changes: * fs: * (SEMVER-MINOR) add `FileHandle.prototype.readLines` (Antoine du Hamel) #42590 * http: * (SEMVER-MINOR) add writeEarlyHints function to ServerResponse (Wing) #44180 * http2: * (SEMVER-MINOR) make early hints generic (Yagiz Nizipli) #44820 * lib: * (SEMVER-MINOR) refactor transferable AbortSignal (flakey5) #44048 * src: * (SEMVER-MINOR) add detailed embedder process initialization API (Anna Henningsen) #44121 * util: * (SEMVER-MINOR) add default value option to parsearg (Manuel Spigolon) #44631 PR-URL: #44968
watch mode (experimental): Running in 'watch' mode using `node --watch` restarts the process when an imported file is changed. Contributed by Moshe Atlow in [#44366](#44366) Other notable changes: * fs: * (SEMVER-MINOR) add `FileHandle.prototype.readLines` (Antoine du Hamel) #42590 * http: * (SEMVER-MINOR) add writeEarlyHints function to ServerResponse (Wing) #44180 * http2: * (SEMVER-MINOR) make early hints generic (Yagiz Nizipli) #44820 * lib: * (SEMVER-MINOR) refactor transferable AbortSignal (flakey5) #44048 * src: * (SEMVER-MINOR) add detailed embedder process initialization API (Anna Henningsen) #44121 * util: * (SEMVER-MINOR) add default value option to parsearg (Manuel Spigolon) #44631 PR-URL: #44968
watch mode (experimental): Running in 'watch' mode using `node --watch` restarts the process when an imported file is changed. Contributed by Moshe Atlow in [#44366](#44366) Other notable changes: * fs: * (SEMVER-MINOR) add `FileHandle.prototype.readLines` (Antoine du Hamel) #42590 * http: * (SEMVER-MINOR) add writeEarlyHints function to ServerResponse (Wing) #44180 * http2: * (SEMVER-MINOR) make early hints generic (Yagiz Nizipli) #44820 * lib: * (SEMVER-MINOR) refactor transferable AbortSignal (flakey5) #44048 * src: * (SEMVER-MINOR) add detailed embedder process initialization API (Anna Henningsen) #44121 * util: * (SEMVER-MINOR) add default value option to parsearg (Manuel Spigolon) #44631 PR-URL: #44968
watch mode (experimental): Running in 'watch' mode using `node --watch` restarts the process when an imported file is changed. Contributed by Moshe Atlow in [#44366](#44366) Other notable changes: * fs: * (SEMVER-MINOR) add `FileHandle.prototype.readLines` (Antoine du Hamel) #42590 * http: * (SEMVER-MINOR) add writeEarlyHints function to ServerResponse (Wing) #44180 * http2: * (SEMVER-MINOR) make early hints generic (Yagiz Nizipli) #44820 * lib: * (SEMVER-MINOR) refactor transferable AbortSignal (flakey5) #44048 * src: * (SEMVER-MINOR) add detailed embedder process initialization API (Anna Henningsen) #44121 * util: * (SEMVER-MINOR) add default value option to parsearg (Manuel Spigolon) #44631 PR-URL: #44968
watch mode (experimental): Running in 'watch' mode using `node --watch` restarts the process when an imported file is changed. Contributed by Moshe Atlow in #44366 Other notable changes: * fs: * (SEMVER-MINOR) add `FileHandle.prototype.readLines` (Antoine du Hamel) #42590 * http: * (SEMVER-MINOR) add writeEarlyHints function to ServerResponse (Wing) #44180 * http2: * (SEMVER-MINOR) make early hints generic (Yagiz Nizipli) #44820 * lib: * (SEMVER-MINOR) refactor transferable AbortSignal (flakey5) #44048 * src: * (SEMVER-MINOR) add detailed embedder process initialization API (Anna Henningsen) #44121 * util: * (SEMVER-MINOR) add default value option to parsearg (Manuel Spigolon) #44631 PR-URL: #44968
watch mode (experimental): Running in 'watch' mode using `node --watch` restarts the process when an imported file is changed. Contributed by Moshe Atlow in #44366 Other notable changes: * fs: * (SEMVER-MINOR) add `FileHandle.prototype.readLines` (Antoine du Hamel) #42590 * http: * (SEMVER-MINOR) add writeEarlyHints function to ServerResponse (Wing) #44180 * http2: * (SEMVER-MINOR) make early hints generic (Yagiz Nizipli) #44820 * lib: * (SEMVER-MINOR) refactor transferable AbortSignal (flakey5) #44048 * src: * (SEMVER-MINOR) add detailed embedder process initialization API (Anna Henningsen) #44121 * util: * (SEMVER-MINOR) add default value option to parsearg (Manuel Spigolon) #44631 PR-URL: #44968
watch mode (experimental): Running in 'watch' mode using `node --watch` restarts the process when an imported file is changed. Contributed by Moshe Atlow in #44366 Other notable changes: * fs: * (SEMVER-MINOR) add `FileHandle.prototype.readLines` (Antoine du Hamel) #42590 * http: * (SEMVER-MINOR) add writeEarlyHints function to ServerResponse (Wing) #44180 * http2: * (SEMVER-MINOR) make early hints generic (Yagiz Nizipli) #44820 * lib: * (SEMVER-MINOR) refactor transferable AbortSignal (flakey5) #44048 * src: * (SEMVER-MINOR) add detailed embedder process initialization API (Anna Henningsen) #44121 * util: * (SEMVER-MINOR) add default value option to parsearg (Manuel Spigolon) #44631 PR-URL: #44968
is there a way to use zlib in this function? or i must use the readline package? |
@kamatil-dev As you can see from the implementation, this is just a thin wrapper around |
@tniessen thank you ❤, but as i can see in my end, the new |
It seems to be a frequent enough use case that it makes sense to have a shortcut method.