Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PostgreSQL output plugin #3428

Open
wants to merge 76 commits into
base: master
from

Conversation

Projects
None yet
@svenklemm
Copy link

commented Nov 5, 2017

This is an output plugin to store metrics in a postgresql database. It will create the tables required if they do not yet exist. Table names are derived from the metric name.

#3408

  • Signed CLA.
  • Associated README.md updated.
  • Has appropriate unit tests.

@russorat russorat modified the milestones: 1.9.0, 1.10 Oct 22, 2018

@markusr

This comment has been minimized.

Copy link

commented Nov 12, 2018

@danielnelson @glinton
Can you merge this? I would like to have this feature.

svenklemm and others added some commits Nov 27, 2018

pg output: Support defining schema for metrics tables
Make sure explicit schema is always used and allow changing that from
public into something else.
pg output: Retry writing metrics after adding missing columns
In some cases metrics datapoints only contain subset of all fields that
may be present for that metric. When storing columns separately instead
of using JSON getting new columns always resulted in the write failing
on first metric that was missing any columns and depending on how often
write was getting called and how many metrics had this behavior there
was potentially very long delay in getting all metrics through.
pg output: Don't try closing nil rows
Used to cause SIGSEGV when the operation failed
@neographikal

This comment has been minimized.

Copy link

commented Jan 5, 2019

Nice work, would like to have this very much :)

@russorat russorat modified the milestones: 1.10.0, 1.11.0 Jan 14, 2019

@sbec67

This comment has been minimized.

Copy link

commented on 404294f Jan 28, 2019

Hello,
is there any further change to get output.postgres official ?
with this Level, i see problem, that for some metrics (f.E. input.system) not all collumns get created:
Regards
S.B

@iojas

iojas approved these changes Feb 15, 2019

@danielnelson danielnelson modified the milestones: 1.11.0, 1.12.0 May 24, 2019

svenklemm and others added some commits May 28, 2019

Refactor PostgreSQL output plugin code
The PostgreSQL plugin code is split up in multiple files for
better readability. Unit and Integration tests are written.
Code complexity reduced a bit by reducing branching.
@blagojts

This comment has been minimized.

Copy link

commented May 30, 2019

Hey guys, we updated the code a bit. Added more unit tests, and created integration tests.
Can you take a look again?

@RobAtticus

This comment has been minimized.

Copy link

commented Jun 14, 2019

Hi @danielnelson @glinton, just wondering if there is any more we can do on our end to move this along?

@danielnelson

This comment has been minimized.

Copy link
Contributor

commented Jun 14, 2019

I'll try to look soon, one question comes to mind now though: what does the plugin do if the same data is written twice?

@blagojts

This comment has been minimized.

Copy link

commented Jun 17, 2019

I'll try to look soon, one question comes to mind now though: what does the plugin do if the same data is written twice?

By default the table being created doesn't have any unique constraints. If the user modifies the table_template config property so that a unique constraint is also added, then the batch containing the duplicate data will fail, subsequent batches won't be affected

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.