Make catalog logging a little quieter where entities are erroring #23022

punkle · 2024-02-16T13:21:41Z

Hey, I just made a Pull Request!

When an error is raised by a url reader or any other processor in the catalog, those errors are logged to the application logs with a warning level as well as written to the refresh_state table for retrieval later.

Given that these errors are already visible at the application layer via the refresh state table, I would argue that the log line is superfluous. The log line can be useful in some cases so I have made it possible to re-enable the log line.

In the end this change makes the catalog logging a little quieter where entities are erroring.

I dont feel terribly strongly about this, and am happy to close this if there is any contention.

✔️ Checklist

A changeset describing the change and affected packages. (more info)
Added or updated documentation
Tests for new functionality and regression tests for bug fixes
Screenshots attached (for UI changes)
All your commits have a Signed-off-by line in the message. (more info)

Signed-off-by: Brian Fletcher <brian@roadie.io>

backstage-goalie · 2024-02-16T13:22:07Z

Changed Packages

Package Name	Package Path	Changeset Bump	Current Version
@backstage/plugin-catalog-backend	plugins/catalog-backend	patch	`v1.21.0`

github-actions · 2024-02-16T13:31:08Z

Uffizzi Cluster pr-23022 was deleted.

awanlin · 2024-02-16T13:31:11Z

For a long time while I was at Keyloop we had no access to the database, I think this is true for other adopters as well. Also, for early Adopters these are errors they will run into first as they learn. I feel like moving them to debug makes them harder to surface especially as turning on the debug logs creates very verbose output making it harder to catch things like this.

That said, happy to go with what every the Maintainer decide on this 👍

punkle · 2024-02-16T14:13:47Z

@awanlin The errors are shown in the application in a few places:

InspectEntityDialog, you can get there from the three dots in the top right of the catalog item.
The plugin gives you an overview of the processing errors also: @backstage/plugin-catalog-unprocessed-entities

mikeyhc · 2024-02-22T16:31:27Z

My concern here is this makes integrating with log based tooling a lot more difficult. We use logs to generate both metrics and alerting, and while this would be possible by enabling debug logging, that would also generate a lot of noise and cost when it comes to storing those logs.

punkle · 2024-02-23T07:38:01Z

Can you elaborate a little? Which metrics and alerts are you triggering from this log line?

mikeyhc · 2024-02-23T15:48:57Z

Sure! We use this to generate metrics and alert around total number of erroring entities. Seeing a steep uptick in this is often indicative of a new automation generating invalid entities, and has helped us proactively reached out to catalog users. Also as a large number of erroring entities can negatively impact processing times it has allowed us to head off incidents related to that.

punkle · 2024-02-26T09:02:47Z

@mikeyhc I suppose what we could do to satisfy both concerns is to emit an event on the event broker and then make a configurable event subscriber that writes a log line or a produces a metric or something..

Then if you dont want the log line, dont configure the subscriber. If you do want the log line, configure the subscriber. Something like that.

Would that work?

Signed-off-by: Brian Fletcher <brian@roadie.io>

mikeyhc · 2024-02-26T14:15:43Z

@punkle sounds good to me

punkle · 2024-02-26T14:26:37Z

@mikeyhc Ive implemented that suggestion in the latest version of this pull request. Let me know if you have any feedback on the specifics.

mikeyhc

LGTM and really appreciate the comprehensive changeset!

github-actions · 2024-03-07T13:43:49Z