What does your post-consume script do?

[theatischbein]

Currently I am working on improving my post-consume script to compensate some features I am missing in Paperless-ngx v2.18.4.

My Questions

• I was wondering, what feature did you implement?
• What are your use cases?
• And maybe do you have some improvements for my setup?

My Usage

I want to have an overview of which invoice I have already worked on and when and improve the payment.

My documents are

• a lot of invoices in german
• different persons as subject (treated person), but same postal recipient (me)
• so many different labels for invoice numbers or customer number (is present at all)

I got an saved view for open invoices

• did not worked using custom fields reliable, so I added an open tag
• submission (for reimbursement) and paid date, help to keep having an overview

My custom field

• submitted date (for reimbursement)
• paid on date
• invoice number
• customer number
• person / patient (choise field)
• bank transfer note (string consisting of person, invoice date, invoice number, customer number)

My post-consume script

Extracting Information

This is the trickiest part.. I tried different things.

• LLMs does work okay, but are way to slow on my machine where paperless is running. Also it does not feel right though
• using nlps like SpaCy for training detecting my data. Problem here is generating the trainings data. My manual annotated documents have the information stored in the custom fields, but I got OCR errors. Something the invoices swap the name to "lastname, firstname" and does not match my custom field. And I dont have the position in the ocr text. This is easy for invoice and customer numbers but not for persons names, because they might appear more often
• using regex my best option but instable to changing labels or new type of invoices

Regex is still the only option I got somehow working.
I created a list of possible patterns, e.g. invoices numbers

Rechungsnummer
Rg.-Nr.
Rechnungsnr.
Rechungs-Nr.
Belegnummer
Belegnr.

and try to match them with seperators like : or spaces/tabs:

pattern = re.compile(
    r"(?:%s)[:\s]+([A-Z0-9/-]+)" % "|".join(re.escape(lbl) for lbl in labels),
    re.IGNORECASE
)

After a while I adjusted for the most typical invoices I got in my monthly/yearly routine.

Filling Custom Fields

The API document is a bit spare concering custom fields.
I got it to work using the endpoint /api/documents/bulk_edit/ and using the parameter field

        resp = sess.post(
            paperless_url +
            "/api/documents/bulk_edit/",
            headers={"Content-Type": "application/json"},
            data=json.dumps(
                {
                    "documents": [doc_pk],
                    "method": "modify_custom_fields",
                    "parameters": {
                        "add_custom_fields": custom_field_data,
                        "remove_custom_fields": [],
                    }
                }),
            timeout=timeout,
        )

I got also a bit tricky I had to do

• a lookup for field names and IDs
• chose/dropdown fields store the data in an encoding it did not recognise test becomes qPjd4A0T7VoNwb5C. I also did a manual lookup for this

Updating Title

I am a bit sad that I need to update the document title via the post-consume script. I really love that there are the custom fields, but it is frustating that I can't use them to set the title via the WebUI.

Editing the title via bulk_edit is not possible and the endpoint /api/documents/{doc_pk} is CSRF protected, so I didn't find a way to update the title via the post_consume script. Sadly!

[github-actions[bot]]

This discussion has been automatically closed due to inactivity. Please see our contributing guidelines for more details.

[github-actions[bot]]

This discussion has been automatically locked since there has not been any recent activity after it was closed. Please open a new discussion for related concerns. See our contributing guidelines for more details.