Allow to configure a seperate db for build logs #5306
Replies: 8 comments 1 reply
-
This is something which is important for our case as well. We have more than 50 teams and 300 over pipelines(it will grow). This will be a potential risk. |
Beta Was this translation helpful? Give feedback.
-
Is it feasible for you to use the existing syslog functionality, coupled with a more aggressive build pruning strategy? Then you could have a brief retention period for "roughly current" builds where they're visible in Concourse itself - including the important case of builds which are currently running - backed by another system, where logs can be retained for longer. Perhaps that's syslog feeding into Filebeat then Elasticsearch, or it's syslog to fluentd to S3, or whatever you fancy. Postgres is honestly not the most natural choice for log storage, but it does have the advantage that it's already there in any Concourse installation - as opposed to an additional dependency for operators to set up and manage. If I imagine a second Postgres for just the logs, I can see how that would work at a code level, and it would have a few benefits. One is that log reaping could happen without necessarily degrading performance on the "main" DB. (Currently, if you have a "really huge" build log from a runaway process, it can be tricky to delete since Postgres doesn't love multi-million-row changes.) But otherwise I am quite wary of this idea since it's a fair amount of extra complexity, all to store logs in basically the same way. Personally, I think one thing that could be handy is the ability to read logs back in from wherever it is that syslog put them. Right now, old builds are tombstoned, but perhaps we could instead arrange for them to gain a "cold storage URL". When you tried to view such a build, Concourse would HTTP That said, this would involve a few extra bits and pieces of configuration.
|
Beta Was this translation helpful? Give feedback.
-
Based on the discussion and Alex inputs, I tried to implement a POC in my fork to store pipeline logs into an Elasticsearch cluster and it works seamlessly. The idea is to provide an optional external storage solution, probably an elasticsearch cluster, for persisting pipeline event logs instead of storing in postgres. |
Beta Was this translation helpful? Give feedback.
-
The support for pluggable event processor looks a great value add. |
Beta Was this translation helpful? Give feedback.
-
@deepakmohanakrishnan1984 if you're referring to my commit here: 1d3172d, just FYI it's very much a WIP, and is just something I'm experimenting with - so I wouldn't expect it any time soon. That said, I'll keep playing around with it when I get the chance, and hopefully it'll materialize into something useful! |
Beta Was this translation helpful? Give feedback.
-
yes, I was referring to your commit and the vision you have on chaining event processors. I will look deeper on it shortly and try if I can contribute fixing some tests or writing a terminal event processor for storing events in elasticsearch(which I am super interested). |
Beta Was this translation helpful? Give feedback.
-
That sounds cool! I'm going to convert this issue to a discussion for now, so hopefully we can get some more visibility on this and ideas from the rest of the community! |
Beta Was this translation helpful? Give feedback.
-
I've added an RFC with a proposal for how to approach this: concourse/rfcs#53 If anyone has the chance to read it through, I'd be happy to hear any feedback! |
Beta Was this translation helpful? Give feedback.
-
What challenge are you facing?
As more and more pipelines are onboard, I see build logs as a potential risk. Some pipelines require to retain very long period of build logs.
What would make this better?
I'm thinking if Concourse can allow a separate database to store build logs. Which has at least two benefits:
Are you interested in implementing this yourself?
Back-end change is ok, but I try to avoid to touch UI code.
Beta Was this translation helpful? Give feedback.
All reactions