Synced files are sometimes unexpectedly truncated

[aaronc]

I notice this issue when an outside process (in this case figwheel) updates files in my projects resources folder. Boot syncs them to temporary files but sometimes the files are unexplainably truncated (i.e. missing the last n lines). I can verify that the file in my actual resources folder is correct and that the file in boot's temp folder is truncated by examining the corresponding file in boot's fileset.

[micha]

Can you describe your setup please? Maybe paste build.boot and project.clj if possible?

[aaronc]

I'm calling figwheel from within a boot pod that's running in a totally
separate boot instance than the boot instance that is serving files. The
figwheel compiler is not really using boot's fileset's but instead working
directly with the disk and writing to the real resources folder. I'll try
to paste a minimal example on Monday.

On Fri, Jun 12, 2015 at 2:27 PM, Micha Niskin notifications@github.com
wrote:

Can you describe your setup please? Maybe paste build.boot and project.clj
if possible?

—
Reply to this email directly or view it on GitHub
#223 (comment).

[micha]

One thing you can try that might solve the problem is increasing the :watcher-debounce time from the default of 10ms to something like 100ms. This is the amount of time boot will wait for things in the filesystem to settle down before starting to copy files.

(merge-env! :watcher-debounce 100)

My guess is that figwheel is writing to a bunch of files sequentially. The filesystem modify event is fired by the OS when the first file is written to. Boot waits 10ms and then makes its filesystem diff against what it has in the fileset and starts to copy. Meanwhile figwheel is writing to other files, introducing a race condition.

I feel like boot could be smarter about how it does the diffing and copying and detect this condition. Recommendations are certainly welcome 😄

[martinklepsch]

@aaronc closing this, reopen if the problem persists.

[aaronc]

Hi @martinklepsch, this issue is definitely not fixed. By the way @micha, my apologies for not responding earlier - I got side-tracked with other things. Changing the :watcher-debounce doesn't help much. In order to work around this, I actually had to switch my ring handler to use exact file-system paths as opposed to resources because the files were just not getting copied correctly.

[martinklepsch]

@aaronc cool, thanks for the feedback. Is there any chance you can provide a reproduction case of some sort?
I sometimes (rarely though) get StreamClosed exceptions about namespaces from dependencies, this sounds like it could be related maybe? There's an open issue about the StreamClosed stuff: #270.

[aaronc]

Hi Martin, I wish I had time to put together a full reproduction, but what I can tell you quickly is that if you try to run something like figwheel in a separate process and then serve the generated js from a ring handler that serves the files based on their resource path, you'll probably eventually end up with the scenario that the files that boot serves are truncated and you'll know this by looking at your browser's debugger. I imagine the StreamClosed exception is related but I can't say for sure. Basically it looks like boot starts copying a file while the external process is still writing the file and then never gets notified of additional changes later on. As @micha was saying, boot probably needs to be smarter about how it handles these change diffs.

[micha]

@aaronc i believe f87f1ea should fix the truncating issue. Can you try with master please?

Edit -- nevermind, the commit is unrelated to this issue.

[micha]

The files generated by figwheel are probably not updated atomically (eg. write to temp file and move to destination). Assuming this is the case I'm not sure how boot can know when it's safe to read a file, because some other process that boot knows nothing about could be writing to it while it's being read. Unless anyone has ideas for how to approach this I think we can close the issue.