Create example Grafana dashboard for Prometheus metrics #74
Comments
|
Hi, I installed this over the weekend and - at least for me - I could not wrap my head around, how the implemented metrics could be graphed in a sensible way. The metrics for a regular backup of 1 repository look like this: I would like to get some kind of alert, if a repo could not successfully be backed up during the last run. Since the runs are in large intervals, any kind of bar or line graph may not bring a lot of value. E.g. I tried just to graph counters, but since they increment in 6 hour intervals and usually my dashboard show more like the last 1-3 hours, I cannot tell, whether there was a successful backup i.e. a step in the graph. Making rate queries do also not make sense for those intervals. I guess, what I am looking for is some kind of Gauge with successful (or failed) number of jobs the last time the "cron" job ran. I am not a Go programmer, so I cannot really help here (but I will have a look later), but I wanted to just share the experience. Thanks for the module anyhow, it is very reassuring to have an additional automated copy of all repos from an own Gitea instance!! Thanks a lot!! Michael. |
|
Hm, looking into the code, I think there is not enough error handling present, but I am not a Go knowledgeable person. I believe, usually one would return a status code from the backup routine, like from the Gitea.Backup() function. But there are only Fatal errors there, which leads then to the whole container terminating. The other non-fatal error would be when the repo no longer exists or cannot be opened. This error is then handled by just skipping over it, but it does not return a true/false/err object back to the caller. I believe the above idea would need a larger modification. But that is just my thinking after looking at the source for 15 minutes or so… |
|
Hi, |
Wow, this would be perfect - thanks for the quick response! Depending on my health over the next days, maybe I find the time to learn more about Go's exception handling and have a go at it myself… At least as an example for gitea… Thanks for this nice piece of software! Cheers, Michael. |
|
@michaelrommel I took a quick look today and changed the local module a bit, so now you can see if a repository was cloned successfully and how long that particular clone took. Is that sufficient for you? PS: I hope you are well and get well soon. |
|
if you want to try it out, just checkout prometheus_improvement and build it. |
|
Hi Andreas, I gave the building a try, but now I am kind of stuck… pickup is running on a linux server in my home network in the docker container. I did not want to pollute the server with additional development tools and therefore attempted to build it on my Mac. I installed Docker Desktop and got so far, that I can build the container. But now the container images is basically local to that Docker Desktop instance. Any idea, how I could get the Linux Server to pull the image from the Mac? Unusual question, I know, but maybe you know the answer off the top of your head. I will be researching on my own as well, how to get the image over… Will keep you posted.. Michael. Edit: |
|
Hm, I guess, I need another linux based build machine… The image on the Mac was build for the linux/arm64 architecture, whereas the server needs a linux/amd64 build. Cross building on the Mac fails with: A task for another day - too late for now... |
|
Got it running. Now I am struggling with a multi source configuration:
Is there an example config available? I noticed the comment at the end of the example config that one should be able to specify "pairs", but no matter which YAML config I try, gickup will fail with a parsing error… |
|
So, most of it works now, I have configured it for now with the same 6h interval. But ideally I would like to increase the interval for the GitHub hosted repos to 2 days or even weekly to be a good citizen… If you can provide guidance for the yml file, that would be great. Here is how I configured the Grafana instance: I used in one graph the
In the next one I have the same metric and another one, another Sum() of
The copies seem to work fine: If that runs smoothly, then all my wishes are already fulfilled - thanks a lot for the quick amendment!! Michael. |
|
yesterday I've built Running gickup with different intervals is pretty simple, either create multiple config files and use both at startup, or put everything in the same file like this: source:
github:
- user: cooperspencer
destination:
local:
- path: ~/backup
structured: true
cron: "0 * * * *"
---
source:
github:
- user: michaelrommel
destination:
local:
- path: ~/backup
structured: true
cron: "5 * * * *" |
|
Seems I almost had it: I had one more line with three dashes at the beginning of the file… Will try that later, thanks!! |
|
I just pushed the latest changes to |
|
It works fine for me! I noticed that there is now a conditional testing for the token, I guess that was the reason why - when I first started out with the gitea conf without a token and only username/password pair - it didn't work for me until I added the token :-) I have changed the Grafana panel to the Bar Gauge style and with a bit of tinkering I now get this display, which is ideal for my dashboard:
It's perfect now for me. Now on to adding the rest of the repos on Github and changing the intervals (I have them running now separately). Perhaps one note would be good that also the prometheus config needs to be in the first definition of "pairs". Thanks Andreas and have a nice evening! |
|
I am glad it works out for you. The changes will be in the newest release. |
|
Hi, thank you very much for the updates to the prometheus metrics available. I would be very interested in replicating the dashboard you created @michaelrommel, are you ok with sharing it ?^^ |
|
Dear @DavidGL17, I apologize for the late reply and wish you a good start into 2024. I wanted to give you as much usable information as possible and therefore extracted the gickup portion out of my main dashboard. You should be able to import that directly into Grafana and then you have to change obviously the names of your hosters according to the gickup config file. I hope you will find this useful. Cheers, Michael. |
|
Hey @michaelrommel No worries, have a happy 2024 too. I managed to import it into my Graphana dashboard and it works perfectly, thank you! |
I'm working on this now but I'm pretty new to Grafana so I'm using this as an opportunity to learn!
The text was updated successfully, but these errors were encountered: