Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add visualization tool to debug image sources #494

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

phillebaba
Copy link
Member

@phillebaba phillebaba commented May 19, 2024

This change will finally empower users to debug Spegel image pulling without having to grok the logs. Over the last year a common question that I receive is "how do I know if Spegel is working?". The install process works sometimes too well which confuses people. While the logs are dynamic and do not tell the whole story of the behavior.

This PR adds a new disabled by default debug page which which will visualize image pull events that occurred on the node. The event tracking will for now occur in memory, and will only run when the feature is enabled. This tool is not meant to be used to debug after the fact errors. We will start off small and revisit this feature a couple of more times in the future to expand it.

  • Track both outgoing and incoming requests
  • Filter outgoing and incoming requests
  • Filter based on images
  • Add handler to metrics server
  • Add option to disable visualizer
  • Document feature

Fixes #24

@phillebaba
Copy link
Member Author

While it works we need some better tool to visualize than graphviz.

image

@phillebaba phillebaba force-pushed the feature/visualization branch 2 times, most recently from 068ce2a to bccd4a2 Compare May 19, 2024 20:33
@bittrance
Copy link
Contributor

bittrance commented May 19, 2024

I looked at event logging yesterday and came up with a similar solution, e.g. not logging HEAD requests and using client IP or peer. The main difference is that in my solution I had:

@@ -181,9 +197,9 @@ func (r *Registry) registryHandler(rw mux.ResponseWriter, req *http.Request) str
        }

        // Request with mirror header are proxied.
-       if req.Header.Get(MirroredHeaderKey) != "true" {
+       if req.Header.Get(MirroredHeaderKey) == "" {
                // Set mirrored header in request to stop infinite loops
-               req.Header.Set(MirroredHeaderKey, "true")
+               req.Header.Set(MirroredHeaderKey, uuid.New().String())
                r.handleMirror(rw, req, ref)
                return "mirror"
        }

The UUID can then be used as the ID in logged events, which will allow connecting events from different spegel peers, effectively working as request tracing.

@phillebaba
Copy link
Member Author

I know that Containerd already adds a traceID to its request headers so if we wanted to use it we could. If someone wants this it is better to add this in a new PR. I just feel like adding more debugging features to the logs is the antithesis of this feature.

@phillebaba phillebaba force-pushed the feature/visualization branch 5 times, most recently from f857e58 to 60f1f23 Compare May 21, 2024 08:31
@phillebaba phillebaba force-pushed the feature/visualization branch 11 times, most recently from 01f5e06 to 7e17c03 Compare May 28, 2024 07:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Debug source of image
2 participants