New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Statistics modules should write to time series DB #1289

Closed
cornelinux opened this Issue Oct 26, 2018 · 3 comments

Comments

Projects
None yet
2 participants
@cornelinux
Copy link
Member

cornelinux commented Oct 26, 2018

Statistics data are currently written to the database table monitoringstats. This table can get filled up quite quickly.

  1. We need a way to rotate this data

  2. We should allow stats modules to write to a separated time series database, that can handle this kind of data in a better way.

@cornelinux

This comment has been minimized.

Copy link
Member Author

cornelinux commented Nov 20, 2018

There are several time series databases around.
Maybe we should add an abstractin layer like the audit module and allow adding other database backends for the statistics.

The first possible module could be the SQLStats.
The module interface could look rather similar to the AuditModule.

We keep the library functions in monitoringstats.py like write_stats, delete_stats, get_stats_keys... but are using the corresponding new MonitoringModule.

After we made this modular, we could look for more preformant time series databases like Prometheus, Wisper or timescale DB.

Finally we should drop the old statistics (lib/stats.py) which is based on the audit log an uses pandas!

@cornelinux cornelinux self-assigned this Nov 20, 2018

@plettich

This comment has been minimized.

Copy link
Contributor

plettich commented Nov 21, 2018

BTW: the tests for lib/stats.py didn't really work since no image was generated.

@cornelinux

This comment has been minimized.

Copy link
Member Author

cornelinux commented Nov 23, 2018

@fredreichbier To be able to write to different time series database we need the app-configuration in some task modules like in the do e.g. of the event counter task module:
https://github.com/privacyidea/privacyidea/blob/master/privacyidea/lib/task/eventcounter.py#L73

First I thought of passing the app-config in the do-method, but then I thought it might be nicer to pass the app-config in the constructor of the class.

But now I end up with problems passing a parameter to a constructer of the task modules in your nice lamda voodoo of TASK_MODULES...

e.g. here:
https://github.com/privacyidea/privacyidea/blob/master/privacyidea/lib/periodictask.py#L59

or here:
https://github.com/privacyidea/privacyidea/blob/master/privacyidea/lib/periodictask.py#L47

It might be also only a problem due to the Mock in the tests. Let's talk.

cornelinux added a commit that referenced this issue Nov 24, 2018

cornelinux added a commit that referenced this issue Nov 29, 2018

Fix some review comments
mostly typos

Working on #1289

cornelinux added a commit that referenced this issue Dec 3, 2018

Fix some review comments
mostly typos

Working on #1289
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment