Custom metadata for documents

[Cantello]

This was already discussed on the old issue board (jonaswinkler/paperless-ng#274) and it is still relevant for paperless-ngx.

There were two models suggested by Jonas about a year ago:

• Custom metadata fields either per document type or globally
• A simple text field, where any data can be written and which is picked up by the search (and which can be shown in the list of documents)

Any preferences here? Jonas wrote back then that implementing the first option would be much more complicated than the second one, although proper custom fields would be more elegant (e.g. the use-case of invoice amount & invoice number).

[skorvek]

I like the concept of custom metadata, but I think if it were to be done, wouldn't the use case be additional filter/sorting criteria that appears in the front-end UI filter bar?

That makes the simple text field rather useless, as the document is already consumed and OCR'ed. Filtering down the document type/correspondent would get you most of the way there. Then any additional searching by the metadata field would be essentially the same as searching the full-text (in the invoice # or amount example). I guess if the contents of the metadata were not in the body of the document already that could be useful. But if that were the approach, I'd rather that simple text field be an unlimited length text blob which could work as essentially a free-text note stapled to the document.

The alternative approach of custom metadata fields would be far more useful for searching/filtering.

Maybe both enhancements for separate use cases?

[Cantello]

You are completely right in that if custom metadata are implemented, one needs to be able to filter according to the metadata, so maybe this is actually two feature requests in one.
That the example that I gave (invoice numbers or amounts) should already be present in the OCRed document is not always true, unfortunately. Sometimes tesseract gets confused by similar-looking numbers/characters and sometimes, there are so many different numbers in the document that finding the right one proves to be difficult. E.g. I would like to total all invoices from a given correspondent in a given time span. Then it would be of no use to have the number somewhere in the text but it needs to be present as a proper metadata. I could always name my documents accordingly (correspondent - invoice - reason - number - amount) but that information could not be as elegantly searched for as custom metadata would be.

So to keep it short - you are completely right, both types of metadata for different use cases!

[mtettke123]

I think as a first step just a custom field "notes" would be great. I would a.e. like to enter something like "paid from account xy on date" to invoices. I would say, even searchable could be optional. For me just a simple "note" field would increase the usefullness!

[ignoreigor]

I agree, for some cases a field for notes, keeping unstructured data, is useful. You do a classical search for a document, find it, and -aaah- there are some notes added. Okay.

For me, custom fields would have other qualities. They keep - more or less- structured data for document-attributes, should be searchable -of course- doing their part to discover (or filter for) documents. But I think the biggest use would arise, when the could mainly be filled automatically by filters (which could also by used to set tags, document-types or correspondents). - but that's another feature request and another disscussion...

[mtettke123]

Yes, I see here two feature requests as well.
Notes (should be really simple) and custom fields (like in docspell, where you can a.e. have a field for the amount of a quote and get statistics over this field - great!)

[sambokai]

I think some sort of fuzzy search would solve this. You could search for e.g. a case number and misinterpreted (OCR) documents would still show up as long as the OCR'd result isn't too far off. I'm thinking 1-3 faulty characters out of 10.

[smseidl]

I'm looking for a field to be able to more easily link a document to a key in another application. One of my main Paperless uses is for Explanation of Benefits (EOB) Scanning. So far, I have been creating a matching TAG in Paperless for unique ID that is created as I enter items in my other app. This is resulting in lots of similar tags. Having another (custom) field to enter that data in would help me in having fewer tags.

[itsdaveit]

I have seen some commercial DMS Systems, they mostly work with custom metadata-fields per document type.

This would be a really mighty approach: One could fill them manually, or maybe use PAPERLESS_POST_CONSUME_SCRIPT to fill them up with data extracted from full text, with some advanced logic and/or regex, or even enrich them with data from external data sources.

I have a thousand ideas what i could do with this, by connecting it to ERPNext instances. (BTW: I`m thinking about creating an app which connects erpnext to paperless)

One technical approach would be to create a additional database-table per document type, in which every metadata field has its own column. So by creation of a metadata field you would also have to decide about the field-type: string, int, float, date for example.

With this approach you could also do effective SQL queries to retrieve some more specific data, then you can with a full-text search.

I`m thinking about something like the "edit Doctype" function from the Frappe Framework

But yes, it`s obviously not trivial to implement.

One easier idea It would be to add one or two text fields:

One could be used to store human readable data AKA notes, the other for storing metadata (key,value pairs) in json, for example.

[TheDarthFather]

• Custom metadata fields either per document type or globally

I belive this is the better design.
Per document type is nice, but not critical.
If is implemented per document type, needs to be as well possible as global ... eitherwise will make the maitenance harder if you have a few you want to create for all the document types .. existent or future.

[hacker-h]

How could the search ui look like for filtering?
Will e.g. the search filters change dynamically to the respective metadata fields if you specify a document type?

[TheDarthFather]

I am not an expert in DMS apps, but we developing at work multiple apps with similar searching capabilities.
The way this type of flexible/custom fields is implemented in most apps I've seen ... is quite simple and dynamic.
The user can select from combo-box the metadata he wants to filter for (ex. DocumentType) and he has a text-box on the same line ... or a combo-box (if we allow only a predefined set of data) .. to fill the metadata value to filter for (ex: Invoice).
He can add another line with metatdata-type / metadata-value-search-for ... until he is happy with how many filters he has ..
This assumes a 'AND' between those predicates.. but some apps allow OR .. and also paratheses ..
This is in general the 'advanced filter' ... the standard and easy to access filter is in general a text, tags or a basic set of metadata available for all [Created Date, Owner, Document Type etc]

If the filter can be saved ... this allows for mutiple saved-views .. and you will not use the advanced filter too much ... but only to define new views .. or with special ocasions [wow where is that document about thats stuff .... I added for my wife ... in that summer .. or the other sumemer :) ]...

[ColinGski]

Hi Everyone,

I recently started looking for a document management system for my private use and I stumbled upon paperless which looks great.
But I couldn't find custom metadatas and I naturally ended up on this discussion.

In a previous job, I developed a small software used internally to classify important amounts of documents. Quite similar to paperless in the idea.
There was custom metadatas on documents and it is for me the most useful feature.
A tag can't have a value and a document type could essentially be a tag.
The best way to properly describe and search a document is through custom metadatas where you can store meaningful information in a structured way.

This has a lot of side benefits like being able to store ids from other systems (customer id, bill id, etc).

The OCR is great but it does not understand the content and it is not always perfectly interpreting the text.
Ideally, you could use the OCR result to try populate custom metadata values based on regular expressions.

[mtettke123]

Any news on this? Would really like to sponsor the easiest implementation (just a "notes" field for ... notes; not mandatory searchable) with a pizza and a beer!

[Cantello]

Over at #687 there have been first attempts at a notes field, you might get involved there.

[aryasenna]

Any news going into this direction? maybe we should raise a Bountysource or something :)

[Cantello]

Comments including search have been implemented now, starting at version 1.12.

[aryasenna]

CMIIW, comment is basically an unstructured, and unvalidated metadata (only single field free text), right?

I think lots of people (me included) rely on custom fields. For me personally it's for future export and parseability.

[JCapriotti]

I've been following this for a while, I'd be interested in contributing to that.

Separately, I wanted to try and implement this, time is always an issue. I'm also a little worried about doing it how I think it should work, and having that be off-base. I'd rather get a minimum viable product out there than do too much.

[pandruszkow]

I wouldn’t be opposed to that, personally. On 6 Apr 2023, at 22:05, Arya Senna ***@***.***> wrote: Any news going into this direction? maybe we should raise a Bountysource or something :) —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[xximj]

+1 For this

[mrmartin95]

Custom metadata would be a great addition. One of the few things I still miss in paperless

[mibe20]

Hello community,

Is ist not possibel to combine 3 of the top feature request merge + reference + metadata?

1. Merge multiple documents into a single one Merge multiple documents into a single one #367 Merge multiple documents into a single one #367
2. Reference between documents Reference between documents #422 Reference between documents #422
3. Custom metadata for document types Custom metadata for documents #141 Custom metadata for documents #141

Make metadata fields which have different typs e.g, Date, referenz like Tags, text, text search with the integatet KI ?

Or would it be possibel to expand the commend field with KI Search options to get special values from document?

Finally use the existing and add some nice stuff like paperless-ngx it already do so much time to paperless-ng.

Thanks to all the greate developers you are incredible.

[schtibal]

Using a custom metadata field for a document reference is not a lot different to using tags or comments. Yes, it can work, but from a usability perspective it requires a lot of customisation to make it intuitive.

There's enough of a difference between how a reference feature should work, and how a custom metadata feature should work, that they should both be implemented differently and independently of eachother.

[xximj]

Hi @schtibal ,

I understand your view on tags and comments, but let's think of metadata as a super-charged tag. This feature is so beneficial for small organizations, enhancing their ability to organize their archive, even with limited resources. Yes, it may require some extra effort to set up initially, but the efficiency it brings to document management make it well worth the investment.

[schtibal]

@xximj you misunderstand me. I'm absolutely for adding custom metadata. And I'm absolutely for adding linking documents. What I am absolutely against, is attempting to use a custom metadata feature to implement document linking.

@mibe20 is suggesting a specific way to use custom metadata fields in order to implement document linking. My comment was merely saying that using custom metadata to implement linking is no better than using comments or tags to implement document linking.

[tjedmunds]

I came across this feature request looking for a solution to my use case which hasn't been mentioned so far but seems related. Among other things I store a lot of receipts for HSA expense tracking. I wanted to store the total dollar amount in a custom field. I presently use a tag for the year and thought it would be cool if I could query answers to questions like 'how much did I spend on tag 'x' (i.e. Medical) in year 2023 (i.e. tag 2023)?' There are a ton of other possibilities for something like this.

[w41ru5]

I would love to have a custom meta field for account number, and batch number for our billing documents. +1

Is there a bounty program for something like this?

[andreheuer]

I would also appreciate having customs fields. For me, a sum (as a an invoice sum) would be really helpful for example. Having look to docspell, there are five types: string, integer (numeric), boolean, date, and currency. I think, these types would cover most of the use cases.

[fivestones]

I'd really appreciate the ability to add a comment or note to something. In Teedy, this is handled with a comments field which show up like a chat about a document when multiple users add comments. They are then indexed and show up in search results.

[stumpylog]

Documents can already have notes on them.

[bertbesser]

@stumpylog do you have a link to the docs, ideally the API docs on reading/writing the notes programmatically? i never found that in the docs ... thanks!

[ndw]

This worked for me. (I ended up turning what I would like to have been custom metadata fields into notes.)

    resp = requests.post(
        f"{HOST}/api/documents/{docid}/notes/",
        json={'note': note},
        headers={"Authorization": f"token {APIKEY}"})

[bertbesser]

Thank you! Much appreciated. I was thinking about just shoving a YAML in there, with the required metadata fields :-) I can now give this a try.

[eliasp]

My use-cases for custom fields would be:

• receipt date of a document (not the creation date): date
• payment deadline for bills: date
• payed: yes/no
• expiration date: date (or timespan like "10 years relative to the creation date")
• external URL: URL (for linking documents with entities in other tools, such as my finance manager)

[ndw]

Just to add my two cents...

I'm considering migrating from a custom solution to paperless. I have several custom metadata fields that I'd like to be able to preserve: GPS location, last modified date, postal address, source URI (where this came from) and "favorite". There are a few others, but they can be implemented as tags. Favorite can probably be a tag as well.

GPS is the only one that looks like it would really benefit from being searchable.

[nimodius]

Look what was just merged! #4502

[github-actions[bot]]

This discussion has been automatically locked since there has not been any recent activity after it was closed. Please open a new discussion for related concerns.