Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

paperless-ng: init at 1.4.5 #123046

Merged
merged 4 commits into from Aug 18, 2021
Merged

paperless-ng: init at 1.4.5 #123046

merged 4 commits into from Aug 18, 2021

Conversation

Flakebi
Copy link
Member

@Flakebi Flakebi commented May 14, 2021

Motivation for this change

“A supercharged version of paperless: scan, index and archive all your physical documents”: https://github.com/jonaswinkler/paperless-ng

Activating services.paperless-ng.enable = true; updates the database, so be sure to make a backup if you used paperless before (the services use the same default directory).

Fixes #116761

Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Ensured that relevant documentation is up to date
  • Fits CONTRIBUTING.md.

@r-rmcgibbo
Copy link

r-rmcgibbo commented May 14, 2021

Result of nixpkgs-review pr 123046 at 016a9013 run on aarch64-linux 1

12 packages built successfully:
  • paperless-ng
  • python38Packages.concurrent-log-handler
  • python38Packages.inotify-simple
  • python38Packages.inotifyrecursive
  • python38Packages.tika
  • python38Packages.watchgod
  • python39Packages.concurrent-log-handler
  • python39Packages.inotify-simple
  • python39Packages.inotifyrecursive
  • python39Packages.tika
  • python39Packages.watchgod
  • xkeysnail
11 suggestions:
  • warning: unused-argument

    Unused argument: stdenv.
    Near pkgs/development/python-modules/inotifyrecursive/default.nix:2:3:

      |
    2 | , stdenv
      |   ^
    
  • warning: unused-argument

    Unused argument: stdenv.
    Near pkgs/development/python-modules/watchgod/default.nix:2:3:

      |
    2 | , stdenv
      |   ^
    
  • warning: no-python-tests

    Test runner could not discover any test cases: ‘Ran 0 tests in 0.000s’
    Near pkgs/development/python-modules/watchgod/default.nix:17:0:

       |
    17 |     description = "Simple, modern file watching and code reload in python.";
       | ^
    
  • warning: no-python-tests

    Test runner could not discover any test cases: ‘Ran 0 tests in 0.000s’
    Near pkgs/development/python-modules/inotifyrecursive/default.nix:20:0:

       |
    20 |     description = "Simple recursive inotify watches for Python";
       | ^
    
  • warning: python-include-tests

    Consider adding a checkPhase for tests, or if not feasible, pythonImportsCheck.

    Near pkgs/development/python-modules/tika/default.nix:24:0:

       |
    24 |     description = "A Python binding to the Apache Tika™ REST services";
       | ^
    
  • warning: unused-argument

    Unused argument: stdenv.
    Near pkgs/applications/office/paperless-ng/default.nix:1:3:

      |
    1 | { stdenv
      |   ^
    
  • warning: unused-argument

    Unused argument: stdenv.
    Near pkgs/development/python-modules/tika/default.nix:2:3:

      |
    2 | , stdenv
      |   ^
    
  • warning: python-include-tests

    Consider adding a checkPhase for tests, or if not feasible, pythonImportsCheck.

    Near pkgs/development/python-modules/inotify-simple/default.nix:17:0:

       |
    17 |     description = "A simple Python wrapper around inotify";
       | ^
    
  • warning: unclear-gpl

    lgpl3 is a deprecated license, please check if project uses lgpl3Plus or lgpl3Only and change meta.license accordingly.

    Near pkgs/development/python-modules/inotifyrecursive/default.nix:22:5:

       |
    22 |     license = licenses.lgpl3;
       |     ^
    
  • warning: no-python-tests

    Test runner could not discover any test cases: ‘Ran 0 tests in 0.000s’
    Near pkgs/development/python-modules/concurrent-log-handler/default.nix:20:0:

       |
    20 |     description = "An additional log handler for Python's standard logging package";
       | ^
    
  • warning: unused-argument

    Unused argument: stdenv.
    Near pkgs/development/python-modules/concurrent-log-handler/default.nix:2:3:

      |
    2 | , stdenv
      |   ^
    

Result of nixpkgs-review pr 123046 at 016a9013 run on x86_64-linux 1

13 packages built successfully:
  • paperless
  • paperless-ng
  • python38Packages.concurrent-log-handler
  • python38Packages.inotify-simple
  • python38Packages.inotifyrecursive
  • python38Packages.tika
  • python38Packages.watchgod
  • python39Packages.concurrent-log-handler
  • python39Packages.inotify-simple
  • python39Packages.inotifyrecursive
  • python39Packages.tika
  • python39Packages.watchgod
  • xkeysnail
11 suggestions:
  • warning: python-include-tests

    Consider adding a checkPhase for tests, or if not feasible, pythonImportsCheck.

    Near pkgs/development/python-modules/tika/default.nix:24:0:

       |
    24 |     description = "A Python binding to the Apache Tika™ REST services";
       | ^
    
  • warning: unused-argument

    Unused argument: stdenv.
    Near pkgs/development/python-modules/concurrent-log-handler/default.nix:2:3:

      |
    2 | , stdenv
      |   ^
    
  • warning: python-include-tests

    Consider adding a checkPhase for tests, or if not feasible, pythonImportsCheck.

    Near pkgs/development/python-modules/inotify-simple/default.nix:17:0:

       |
    17 |     description = "A simple Python wrapper around inotify";
       | ^
    
  • warning: unused-argument

    Unused argument: stdenv.
    Near pkgs/development/python-modules/tika/default.nix:2:3:

      |
    2 | , stdenv
      |   ^
    
  • warning: no-python-tests

    Test runner could not discover any test cases: ‘Ran 0 tests in 0.000s’
    Near pkgs/development/python-modules/concurrent-log-handler/default.nix:20:0:

       |
    20 |     description = "An additional log handler for Python's standard logging package";
       | ^
    
  • warning: no-python-tests

    Test runner could not discover any test cases: ‘Ran 0 tests in 0.000s’
    Near pkgs/development/python-modules/watchgod/default.nix:17:0:

       |
    17 |     description = "Simple, modern file watching and code reload in python.";
       | ^
    
  • warning: no-python-tests

    Test runner could not discover any test cases: ‘Ran 0 tests in 0.000s’
    Near pkgs/development/python-modules/inotifyrecursive/default.nix:20:0:

       |
    20 |     description = "Simple recursive inotify watches for Python";
       | ^
    
  • warning: unclear-gpl

    lgpl3 is a deprecated license, please check if project uses lgpl3Plus or lgpl3Only and change meta.license accordingly.

    Near pkgs/development/python-modules/inotifyrecursive/default.nix:22:5:

       |
    22 |     license = licenses.lgpl3;
       |     ^
    
  • warning: unused-argument

    Unused argument: stdenv.
    Near pkgs/applications/office/paperless-ng/default.nix:1:3:

      |
    1 | { stdenv
      |   ^
    
  • warning: unused-argument

    Unused argument: stdenv.
    Near pkgs/development/python-modules/inotifyrecursive/default.nix:2:3:

      |
    2 | , stdenv
      |   ^
    
  • warning: unused-argument

    Unused argument: stdenv.
    Near pkgs/development/python-modules/watchgod/default.nix:2:3:

      |
    2 | , stdenv
      |   ^
    

Copy link
Member

@dotlambda dotlambda left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should add an assertion that paperless is not enabled when paperless-ng is.

@Flakebi
Copy link
Member Author

Flakebi commented May 15, 2021

We should add an assertion that paperless is not enabled when paperless-ng is.

Good idea, I added an assertion if they both point to the same data directory.

@erikarvstedt
Copy link
Member

erikarvstedt commented May 15, 2021

I added an assertion if they both point to the same data directory

Both services still use the same port. We should change the default paperless-ng port from 28981 to 28982 or extend the assertion.

@Flakebi, could you address the stdenv warnings in #123046 (comment)?

@Flakebi
Copy link
Member Author

Flakebi commented May 15, 2021

Extended the assertion to check the port and fixed the warnings.

Copy link
Member

@dotlambda dotlambda left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Most commit messages should start with pythonPackages or python3Packages. All Python packages should have explicit tests, i.e. set checkPhase or use pytestCheckHook (unless you set doCheck = false for some reason), and use pythonImportsCheck.

You might also want to look into doing hardening the systemd unit. Take inspiration from other modules or read https://www.freedesktop.org/software/systemd/man/systemd.exec.html#Sandboxing, https://www.freedesktop.org/software/systemd/man/systemd.resource-control.html#IPAddressAllow=ADDRESS%5B/PREFIXLENGTH%5D%E2%80%A6, and https://www.freedesktop.org/software/systemd/man/systemd.resource-control.html#DevicePolicy=auto%7Cclosed%7Cstrict.

nixos/modules/services/misc/paperless-ng.nix Outdated Show resolved Hide resolved
nixos/modules/services/misc/paperless-ng.nix Outdated Show resolved Hide resolved
nixos/modules/services/misc/paperless-ng.nix Show resolved Hide resolved
Copy link
Member

@erikarvstedt erikarvstedt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I haven't yet finished my review, but to avoid duplications in @dotlambda's and my review, here are my fixes so far.

@Flakebi
Copy link
Member Author

Flakebi commented May 15, 2021

Thanks for the helpful reviews!

@erikarvstedt, I added all the fixes from your branch.

All new python packages now have pythonImportsCheck and the systemd services should be a bit more secure now (systemd-analyze security is quite useful for that).

Copy link
Member

@erikarvstedt erikarvstedt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Flakebi, thanks for packaging paperless-ng!
Here are some more fixups.

@Flakebi
Copy link
Member Author

Flakebi commented May 16, 2021

Thanks for the fixups @erikarvstedt!

I kept the TemporaryFileSystem. I found that somewhere in nixpkgs (although I can’t remember where) and I like it quite a bit because it also hides /etc and the likes.

@ThibautMarty
Copy link
Member

Just a thought: how about using services.paperless instead of services.paperless-ng? If I understood correctly the services are not intended to be run in parallel (same data directory, same port, etc. and more importantly same service). Even if it is possible, that would be a corner case I guess.

Are people expected to use the old paperless? Are the modules interfaces similar?

If the answers are no and yes, we could use the paperless-ng module+package if system.stateVersion is new enough. Just like a database engine update needing migration.

Copy link
Member

@erikarvstedt erikarvstedt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Flakebi
Copy link
Member Author

Flakebi commented Jun 22, 2021

Rebased to fix merge conflicts

@ofborg ofborg bot requested a review from erikarvstedt June 22, 2021 20:28
Copy link
Member

@erikarvstedt erikarvstedt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@andir, can you help getting this merged?

@andir
Copy link
Member

andir commented Jun 24, 2021

@ofborg test paperless-ng

@alarsyo
Copy link
Contributor

alarsyo commented Jul 8, 2021

Thanks for working on this! Just a heads up, looks like the latest release is now 1.4.5 :)

(edit: I just saw that you did update the version, PR title and commit still mention previous one)

@Flakebi Flakebi changed the title paperless-ng: init at 1.4.4 paperless-ng: init at 1.4.5 Jul 8, 2021
@Flakebi
Copy link
Member Author

Flakebi commented Jul 9, 2021

Rebased to fix merge conflicts and corrected commit message to 1.4.5.
The test is currently failing due to the last fontconfig update to 2.13.93. It will start working again once 2.13.94 is merged into master.

@ofborg ofborg bot requested a review from erikarvstedt July 9, 2021 21:01
@alarsyo
Copy link
Contributor

alarsyo commented Jul 9, 2021

Do you think this could get backported to 21.05 once it gets merged, without the commit removing paperless? I'd be interested in running this on stable :)

@SuperSandro2000
Copy link
Member

SuperSandro2000 commented Jul 9, 2021

Do you think this could get backported to 21.05 once it gets merged, without the commit removing paperless? I'd be interested in running this on stable :)

It may be that easy or it may be not. If we need to backport more things I don't think we should do it at all and generally I don't think we should backport new additions anymore.

@ambroisie
Copy link
Contributor

fontconfig seems to have been updated, @Flakebi friendly ping :-)

@Flakebi
Copy link
Member Author

Flakebi commented Jul 22, 2021

Rebased to fix merge conflicts and fix test failures caused by Pillow update.

@Flakebi
Copy link
Member Author

Flakebi commented Jul 26, 2021

Rebased and slightly improved security by removing ProtectSystem=strict, which re-mounted all root folders read-only into the temporary root directory. Without ProtectSystem=strict, the folders are not visible at all.

Comment on lines +168 to +172
checkPhase = ''
pushd src
PATH="${path}:$PATH" HOME=$(mktemp -d) XDG_DATA_DIRS="${liberation_ttf}/share:$XDG_DATA_DIRS" pytest
popd
'';
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should use pytestCheckHook instead.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As discussed before, pytestCheckHook introduces needless complexity by requiring an extra postCheck hook for restoring the working directory. Using checkPhase is shorter and clearer.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As soon as we need to disable some test or directory of tests pytestCheckHook is way cleaner and easier to use. The extra postPhase does not matter. If we have hooks and functions we should use them instead of reimplementing stuff.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@SuperSandro2000, I'd refer to YAGNI in this case and steer on the side of explicitness and simplicity.

@andir, would you consider merging this PR in its current state, given the merge conflict is resolved? I'd really love to see this getting finished.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@erikarvstedt Yeah, I am looking forward to having this in nixpkgs so I can stop rebasing :)

@ntdef
Copy link

ntdef commented Aug 5, 2021

I would also very much like to see this get merged! I've had a difficult time trying to incorporate this package into my NixOS.

For what it's worth, I've tried to run nix build on this package, and the tests are failing for me. I ran:

nix build 'github:Flakebi/nixpkgs/32095dd69ca8af3fd55e2224b60e21a0ae074a8a#paperless-ng'

And I get:

[ntdef@nixos ~]$ nix build github:Flakebi/nixpkgs/32095dd69ca8af3fd55e2224b60e21a0ae074a8a#paperless-ng 
error: builder for '/nix/store/y4nnyrqzf98i5qs1mlnmgwxycd1a8ik0-paperless-ng-1.4.5.drv' failed with exit code 1;
       last 10 log lines:
       > =========================== short test summary info ============================
       > FAILED documents/tests/test_management_consumer.py::TestConsumerPolling::test_slow_write_pdf
       > FAILED documents/tests/test_management_consumer.py::TestConsumerRecursivePolling::test_slow_write_pdf
       >
       > Results (97.77s):
       >      427 passed
       >        2 failed
       >          - documents/tests/test_management_consumer.py:134 TestConsumerPolling.test_slow_write_pdf
       >          - documents/tests/test_management_consumer.py:134 TestConsumerRecursivePolling.test_slow_write_pdf
       >        2 skipped
       For full logs, run 'nix log /nix/store/y4nnyrqzf98i5qs1mlnmgwxycd1a8ik0-paperless-ng-1.4.5.drv'.

Here are the full logs:
log.txt

Flakebi and others added 4 commits August 14, 2021 10:10
…less-ng

The paperless project has moved on to paperless-ng and the original
paperless package in Nixpkgs has stopped working recently (due to
version incompatibility with the providede Django package).

Instead of investing more time into the old module we should migrate all
users to the new module instead.
@andir
Copy link
Member

andir commented Aug 14, 2021

@ofborg test paperless-ng

@erikarvstedt
Copy link
Member

@andir, the test succeeds. Is there anything left to be done?

@andir andir merged commit ae61a14 into NixOS:master Aug 18, 2021
@Flakebi Flakebi deleted the paperless branch August 18, 2021 12:49
@Flakebi Flakebi mentioned this pull request Sep 24, 2021
12 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

paperless: out-of-date / upstream archived