Feature: consumption templates

[shamoon]

Proposed change

This PR adds "consumption templates" which allow users more control over consumed docs. Functionally, this allows things like automatically sharing new documents, different consume folders and other useful things.

In general, this could definitely be expanded upon in the future, I think.

TLDR:

• Set document source (folder, API, mail) and filter docs by file name, path or mail rule
• Allows assigning: title (with some placeholders), tags, correspondent, doc type, owner & permissions

Thoughts:

• At the moment templates will be [potentially] processed sequentially but subsequent templates won't override the prior. e.g. if you have two templates that set the correspondent the first one applied 'wins'.
• Title placeholders have to come from the template so currently {correspondent}, {document_type}, {owner_username} and some date variables for added.
• I thought about migrating mail rules to this (you add a template to a mail rule) but in the end I left them as-is, it seemed nice to be able to have two places to control things. Mail rule settings (e.g. assign correspondent) will take precedence.
• I allowed filtering by mail rule, that way you can apply a template to docs consumed via a rule (to do things like set permissions, etc).
• I also modified mail rules to allow a) not setting title, b) not setting owner

TODO:

• docs
• backend testing

Fixes #2177
Fixes #4163
Fixes #3443
& others

Type of change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Other (please explain):

Checklist:

• I have read & agree with the contributing guidelines.
• If applicable, I have included testing coverage for new code in this PR, for backend and / or front-end changes.
• If applicable, I have tested my code for new features & regressions on both mobile & desktop devices, using the latest version of major browsers.
• If applicable, I have checked that all tests pass, see documentation.
• I have run all pre-commit hooks, see documentation.
• I have made corresponding changes to the documentation as needed.
• I have checked my modifications for any breaking changes.

[codecov]

Codecov Report

Merging #4196 (19188bb) into dev (86d223f) will increase coverage by 0.16%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##              dev    #4196      +/-   ##
==========================================
+ Coverage   95.45%   95.61%   +0.16%     
==========================================
  Files         344      352       +8     
  Lines       13244    13571     +327     
  Branches     1084     1088       +4     
==========================================
+ Hits        12642    12976     +334     
+ Misses        597      590       -7     
  Partials        5        5              

Flag	Coverage Δ
backend	94.47% <100.00%> (+0.25%)	⬆️
frontend	96.97% <100.00%> (+0.06%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files Changed	Coverage Δ
.../app/components/app-frame/app-frame.component.html	100.00% <ø> (ø)
...e-edit-dialog/mail-rule-edit-dialog.component.html	100.00% <ø> (ø)
...ule-edit-dialog/mail-rule-edit-dialog.component.ts	100.00% <ø> (ø)
...g/user-edit-dialog/user-edit-dialog.component.html	100.00% <ø> (ø)
...p/components/common/input/tags/tags.component.html	100.00% <ø> (ø)
src/paperless_mail/serialisers.py	97.95% <ø> (ø)
src-ui/src/app/app-routing.module.ts	100.00% <100.00%> (ø)
src-ui/src/app/app.module.ts	100.00% <100.00%> (ø)
...og/consumption-template-edit-dialog.component.html	100.00% <100.00%> (ø)
...alog/consumption-template-edit-dialog.component.ts	100.00% <100.00%> (ø)
... and 19 more

... and 1 file with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

[stumpylog]

This looks very powerful for the permissions and just other stuff. I'll have to look at it more when I can. It seems like a good answer to many requests overall. Nice work so far.

[shamoon]

This looks very powerful for the permissions and just other stuff. I'll have to look at it more when I can. It seems like a good answer to many requests overall. Nice work so far.

Thanks mate, yea whatever (/whenever) you can contribute would be awesome, you know I tend to fumble through backend stuff

[ignoreigor]

Hi again,

This looks very powerful for the permissions and just other stuff. I'll have to look at it more when I can. It seems like a good answer to many requests overall. Nice work so far.

i agree this looks powerful and fairly useful for multiple purposes. Maybe this can end up in far future to a complete rewrite of matching document-types, correspondents and tags, then working with a list of rules consisting of conditions and actions, as I earlier suggested.

But so far, good work!

[stumpylog]

The header of the templates page is hard to see on a dark theme:

Tags dropdown doesn't have a "No items found" thing.

After selecting consume and API upload sources, the resulting template always has mail fetch instead. The PUT values look good, the GET returns Mail Source.

I can help track some of this down again as I have time.

[shamoon]

Thanks for the fixes / cleanup (yea all that int() stuff felt messy). Let me know what else you notice.

The column heading thing was from that last package bump (bootstrap release notes are horrible) but fixed, as for the tags dropdown, you mean like this (screenshot below) or something else?

[shamoon]

Turns out you were right stumpy, the default perms on the frontend would apply to these, which got me thinking about this some more. Actually a lot more...

I debated this a bit, but I think the best thing is to remove perms from templates (not assigning perms to docs, just no owner / perms on templates themselves). Happy to come up with another solution if you think wise.

The biggest issue, I think, is that since templates are potentially applied to sources that have no user in context, so listing the templates would always return all of them (as opposed to only ones they have access to) and so you'd end up with users potentially having their docs affected by templates they cant see. Then theres the fact that sharing is kind of nonsensical (so I considered removing perms but leaving owner, but still previous issue), etc. etc.

Overall, seemed too messy and counter-intuitive and I didn't see an obvious simple solution. Again, welcome to thoughts but this way theres one master list and people can either access it, or not.

[stumpylog]

Yes, i see the problem. A user would be surprised if they had a document with unexpected tags or even the owner changed from themselves. This seems like a good compromise

[stumpylog]

I think I'm good with this as it. Lets squash and merge it

[juergen852]

I think, this feature is more something for the "Admins" and not for normal users.
I would also feel good with your approach if templates are limited to admins.

[shamoon]

You can set permissions for this just like anything else in the app

[github-actions]

This pull request has been automatically locked since there has not been any recent activity after it was closed. Please open a new discussion or issue for related concerns.