Fix: Delay consumption after MODIFY inotify events

[frozenbrain]

Proposed change

Fujitsu recently released a firmware update for their iX1600 scanner allowing you to scan directly to a network folder without the need for a PC. Currently Paperless delays consumption after CLOSE_WRITE and MOVED_TO inotify events, however Fujitsu's implementation of network scanning seems to open and close the PDF file multiple times (with a longer open-write-close phase at the end) which causes Paperless to try to consume the file prematurely.

This PR proposes to add the MODIFY event to the list of delaying inotify events.

To better visualize the issue, I've attached two logs of inotifywait during a scan of the same (physical) document. I have PAPERLESS_CONSUMER_INOTIFY_DELAY set to 3 seconds in my Paperless installation. In before.txt you can see Paperless trying to read the file (L122) 3 seconds after the last CLOSE_WRITE event (L76) while the scanner is still writing data. In after.txt you can see Paperless correctly waiting until the file has been fully written.

Type of change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Other (please explain):

Checklist:

• I have read & agree with the contributing guidelines.
• If applicable, I have included testing coverage for new code in this PR, for backend and / or front-end changes.
• If applicable, I have tested my code for new features & regressions on both mobile & desktop devices, using the latest version of major browsers.
• If applicable, I have checked that all tests pass, see documentation.
• I have run all pre-commit hooks, see documentation.
• I have made corresponding changes to the documentation as needed.
• I have checked my modifications for any breaking changes.

[paperless-ngx-secretary]

Hello @frozenbrain,

thank you very much for submitting this PR to us!

This is what will happen next:

1. My robotic colleagues will check your changes to see if they break anything. You can see the progress below.
2. Once that is finished, human contributors from paperless-ngx review your changes. Since this seems to be a small change, only a single contributor has to review your changes.
3. Please improve anything that comes up during the review until your pull request gets approved.
4. Your pull request will be merged into the dev branch. Changes there will be tested further.
5. Eventually, changes from you and other contributors will be merged into main and a new release will be made.

Please allow up to 7 days for an initial review. We're all very excited about new pull requests but we only do this as a hobby.
If any action will be required by you, please reply within a month.

[shamoon]

Thanks for the PR. Makes sense to me but I’ll leave this one for the backend folks

[shamoon]

Ps. I’d be fine with this going into beta

[stumpylog]

Based on the test results, this is adding the file multiple times to the task queue, not just once.

[frozenbrain]

I just realized that I ran my local tests in the wrong branch, sorry about that.

Anyways you're right @stumpylog, MODIFY events now add the files to the consumption queue before the file is actually closed which isn't much better (or arguably even worse).

I've modified the PR so that now files waiting to be consumed are removed from the notified_files dict after MODIFY events. This should delay consumption until after the next CLOSE_WRITE event.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (07c38d6) 96.01% compared to head (a6903d6) 96.01%.

Additional details and impacted files

@@           Coverage Diff           @@
##              dev    #4626   +/-   ##
=======================================
  Coverage   96.01%   96.01%           
=======================================
  Files         371      371           
  Lines       14153    14155    +2     
  Branches     1110     1110           
=======================================
+ Hits        13589    13591    +2     
  Misses        560      560           
  Partials        4        4           

Flag	Coverage Δ
backend	94.65% <100.00%> (+<0.01%)	⬆️
frontend	97.59% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[stumpylog]

Looks fine to me now. I've targeted this to beta instead

[github-actions]

This pull request has been automatically locked since there has not been any recent activity after it was closed. Please open a new discussion or issue for related concerns.