Metrics endpoint for Prometheus 📈

[pascaliske]

Hello maintainers!

Thanks for your awesome work and for taking over the project from the previous maintainer! 🚀

Recently, I thought how cool it would be if paperless would expose metrics for Prometheus. This would enable us to visualize statistics about our documents in beautiful dashboards (e.g. Grafana). A few ideas of possible metrics in my head are:

• count of documents in total
• count of documents in inbox
• count of documents by correspondent and/or by type
• file tasks with state (queued, started, completed, failed)

Prometheus provides an official client library for integrating it into python projects: https://github.com/prometheus/client_python.
There is even a Django compatible library which provides basic metrics from Django out of the box and integrates well.

What do you think? Would this be an interesting idea for the project? If I find some free time, I can even try to build a first proof-of-concept in a PR... See draft PR here: #1544 🙂

[stumpylog]

Is this something which needs to be integrated into paperless itself? To me, it sounds like a great companion, using the API to generate the stats and send them where ever. Potentially, this could be changed from polling to event driven with something like apprise POSTing a JSON string to the companion.

[pascaliske]

Yes, usually this is integrated.

The app provides an HTTP endpoint (e.g. /metrics) which gets scraped/called by prometheus. The endpoint just echoes out the metrics with labels and values in a specific format. It looks something like this:

# HELP paperless_build_info Current version number and build information.
# TYPE paperless_build_info gauge
paperless_build_info{build_time="2022-08-23T08:09:20Z",git_commit="xxxxxx",python_version="3.9",platform="linux/arm",version="1.8.0"} 1
# HELP paperless_config_last_reload_failure UNIX timestamp of last failed config file reload.
# TYPE paperless_config_last_reload_failure gauge
paperless_config_last_reload_failure 0
# HELP paperless_config_last_reload_success UNIX timestamp of last successfull config file reload.
# TYPE paperless_config_last_reload_success gauge
paperless_config_last_reload_success 1.6612453667600596e+09

It would be possible to build an exporter on its own but this would limit the possible metrics to the data which is currently available by the API. Also in my opinion it is cleaner if paperless can expose the metrics directly.

I hope this clarifies it a bit. 🙂

[pascaliske]

I prepared a draft PR as proof-of-concept: #1544. It implements django-prometheus and provides basic metrics for Python and Django out of the box.

Additionally, I will try to implement some example metrics like paperless_documents_total which tracks the total count of documents to better explain how a possible integration could look like.

[sukisoft]

When dealing with counts of documents, maybe #1283 could be something which could be taken into account. Count of documents in inbox (with specific tag) could be very similar to the count of saved views.

[dast5022]

Using API as source for some metrics was also my first idea. But then I recognized, that it would be necessary to give full reading access to all documents to that technical user, that will call the api. In my setting, this wouldn´t be a good idea. Or does I misunderstand something. So I would also be happy to have a special endpoint which will only deliver metrics, no content.

[smseidl]

I've just started playing with Prometheus and Grafana and think this is a great idea to add to PaperlessNGX. Other items that would be nice to see gathered are task status - especially the failed tasks (similar to the new menu item by logs). I hope to take a look at the PR at some time and might work on a dashboard for it.

[hendkai]

Would be great to be intigrated in the api. I created a node red flow for my ulanzi clock. :)

[jan-di]

For others to come up here from a search:

There is third party exporter that provides metrics for paperless-ngx: https://github.com/hansmi/prometheus-paperless-exporter

[tofuwabohu]

I was about to write a similar feature request. The third party exporter is cool, but it can't work to it's full potential without a dedicated metric endpoint.

Monitoring your paperless-ngx stack is a good way to make sure you notice when things happen that shouldn't happen.

Currently, the API supports to view some numbers but all of them are based on what a monitoring user can see. Want to get the total number of documents, users or tags? The monitoring user needs to get superuser permission. If you use a tool like the prometheus-paperless-exporter in order to provide the metrics for your monitoring system, that third party tool has to get superuser access.

My idea is to have /api/metrics, which is bound to it's own permission and would mostly return the total numbers of all the other andpoints, without the need of having the superuser permission. Third party tools can then pull that endpoint and convert it to the metric format they need without superuser permissions.

Bonus:
Some numbers that would be of interest besides "total of other endpoints" like log_events_last_5_minutes grouped by level (e.g. to get an alert if there's a unusual spike in error logs)

[andygrunwald]

Hey Paperless-Team (@bauerj, @jonaswinkler, @shamoon, @stumpylog and @tribut),
I may wanna a native Prometheus endpoint inside Paperless a try. Before I am spending time and effort on this, my question to you:

Is this something in your interest with a possible chance to get into main of Paperless? Or do you prefer to lean on community tools like https://github.com/hansmi/prometheus-paperless-exporter to fulfill this need?

This discussion has a bit of insight on why the native integration might be superior over the external REST-API one. Your voice would decide on me/us/the community spending time on this or not.

Thanks in advance.

[shamoon]

I don’t have very strong feelings about this, for something that is generally just “nice to have” I’m not really sure it’s worth this being part of the paperless core, but I suppose it really depends on how much new code the change introduces, or more specifically the maintenance burden of that. Perhaps there’s also a less invasive way to do this (some kind of special user or privilege that would better facilitate the external project above). But yea especially if it’s opt-in, maybe it’s not a big deal.

[github-actions[bot]]

This discussion has been automatically closed due to lack of community support. Please see our contributing guidelines for more details.

[github-actions[bot]]

This discussion has been automatically locked since there has not been any recent activity after it was closed. Please open a new discussion for related concerns. See our contributing guidelines for more details.