Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Watching/notifications #72

Open
rektide opened this issue Jul 23, 2019 · 8 comments

Comments

@rektide
Copy link

commented Jul 23, 2019

A huge huge huge part of file-system management & capabilities as we use them is being notified about when something changes. Without this, awful awful awful terrible badly performing highly costly & deeply inadequate hacks grow up like weeds, all over the place, as bad terrible coders do an awful job of probing around to figure out "what has changed?". Huge amounts of engineering effort have gone into trying to tackle this issue. Works like node-watch (attempting to fill the painful dx gaps in node's fs.watch being non recursive) & watchman grow like weeds, consuming developer year after developer year of time to make maintain & sustain.

I love the works proposed here, DEARLY, and think they are so on target & in the right direction. But if we don't ship in 1.0 some kind of reactivity, some kind of file watching, given what I see on the rest of the web, it will literally never ever happen, and we will be very very very sad with the years of strife suffering misery & terrible problems that were so easily avoided had anyone bothered to stake in on caring about a moderately successful go at what file system apis need to be & do.

Please please please please please. Please make native-file-system api have watching capabilities. Please let me look at at a subdirectory tree, let me say, if anything in /foo/bar changes, let me know. Without this, the amount of polling & state checking will boil away the oceans, be the most tragic waste of our planet we could imagine. If this is included, we can blissfully begin to carefully slowly rebuild the web in a way that coherently integrates the user experience with the stored shared mutable state that is the filesystem. Being able to see & observe those mutations is a crucial capability that I would put much on the line (were I aware of any such means to) to bring forth to the web, to native-file-system, as it turns this from a dumb, dead, passive tool into something involved, dynamic, & reactive.

Please kindly include some kind of fs-watch like capability.

@pwnall

This comment has been minimized.

Copy link
Collaborator

commented Jul 23, 2019

Thank you for your feedback! Glad you care about this API! We also think it's a very important capability for the Web.

Watching isn't going to make it for 1.0. Implementing it would take a lot of work for a browser, so holding back 1.0 for watching would delay all other use cases that can benefit from what we have today. I understand that watching is very important to you, and it has some applications that I care about, but I wouldn't be fair to ask everyone else to wait while this happens.

If you're interested in making this happen, you could help by putting together a Chrome CL that shows how to implement watching well on Windows, macOS, and Linux. This would help assess how difficult it would be to implement this in a cross-platform browser -- my intuition is that it'd take ~2 engineering years, and I'd really like to be proven wrong.

@guest271314

This comment has been minimized.

Copy link

commented Jul 24, 2019

Please let me look at at a subdirectory tree, let me say, if anything in /foo/bar changes,

Does that definition of "changes" mean appending or removing a single character at, that is, any .txt file outside of Native File System methods being used to do so; for example a text editor changes a file that Native File System has permissions to read/write? Would changes to permission of the file performed at shell also be considered a change; e.g., a file permission being changed to read-only and/or write/execute during permission for Native File System being granted?

@rektide

This comment has been minimized.

Copy link
Author

commented Jul 28, 2019

@pwnall The only thing that turns up when I search for "chrome cl" is this link to something in ChromiumOS, and there's not enough context on that page for me to understand enough to try to follow along. I don't see anything in the Developer Guide that seems very promising. I was thinking perhaps (https://chromium.googlesource.com/chromiumos/docs/+/master/developer_guide.md#Making-changes-to-packages-whose-source-code-is-checked-into-Chromium-OS-git-repositories) was going to give me something to chew on, but it doesn't seem to match up with anything in I see on the "create a cl" page? What was your ask again?

As for your ask for a cross-platform PoC, I have no access to Windows or Mac systems. I've also never worked on Chrome before. It probably would take me two man years to do, sounds about right.

I do want to help but this is also terrifying, and feels a little bit like being sent to never never land.

@rektide

This comment has been minimized.

Copy link
Author

commented Jul 28, 2019

@guest271314 yes to all of the above. I am a bit of a file watching maximalist.

The filesystem is our shared source of truth in computing, and being able to observe the truth happening is important.

@guest271314

This comment has been minimized.

Copy link

commented Jul 28, 2019

@rektide A "a non-terminating procedure that happens to refer to itself" pattern which reads the file continuously? A copy of one or more files would need to be stored, correct? Consider the case of same file open in several tabs in a text editor, or multiple text editors referencing the same file; how to determine which file is the original? Timestamps, or is the original file not important, only indexes of the number of files open having the same path and name and differences between the files? How would the developer be able to process all changes to files in an event-based or Promise based pattern in such a case, that is, the same file potentially being changed multiple times based on input from multiple programs accessing the file? Have not tried reading read-only files in some time; what should occur when a file permission is set to read-only and permission to read the file metadata is requested?

@guest271314

This comment has been minimized.

Copy link

commented Jul 28, 2019

@rektide A DirectoryObserver and FileObserver, similar to PerformanceObserver, IntersectionObserver, and MutationObserver?

@guest271314

This comment has been minimized.

Copy link

commented Jul 30, 2019

Since the user must know the directories and files which potentially will be changed by a write procedure, the program can create a ReadableStream to read every byte of an instance of let stream = writer . asWritableStream(), or similar directory and file writing stream. During the write, notify the changes made to a copy of the original file, then delete the copy, the changed file taking the place of the original file reference.

Code to implement checking for changes made during writing to an existing file and notification code is not an issue.

Notification of changes to permissions of files could begin be creating a time range in which the notification should occur, to avoid a "a non-terminating procedure that happens to refer to itself" pattern relevant to asynchronous code running continuously, consuming computational resources, even when no permissions are being changed during every call to the procedure. For example, from now until then check if directory and/or file permissions changed. Or, a method that checks only when called, not continuously.

How many times should a notification occur? Once, when single instance of write/read procedure completes? Or should notification occur once for each and every change to underlying data source; and directory and file permissions?

@pwnall pwnall added this to the V2 milestone Aug 1, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.