-
Notifications
You must be signed in to change notification settings - Fork 3
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
global-query #43
Comments
I will overwrite the branch from #43 and overwrite it. Too much changed under the hood. I've saved the relevant bits in Way to go about this:
|
Will start progress next week |
Saw https://github.com/els0r/goProbe/actions/runs/4613869362/jobs/8156300674, which seems unrelated to this issue, but still needs to be investigated. The problem is a race condition in line 273 of the DBWorkManager
That number is also accessed in line 256 with:
|
@fako1024 : need help with testing the very first version of
Then, run queries with the tool with go run cmd/global-query/main.go \
--config cmd/global-query/local-config.yaml \
--hosts.querier.config cmd/global-query/api-client-querier.yaml \
-q <comma-separated-host-list> <goquery args> |
Raised #94 and will take care of the race condition. As for the global query I'll gladly give it a shot today. ❤️ |
@els0r OK, first attempt: Deployment went fine, nodes are reachable. However there seems to be an issue with the encoding of the query (probably with the attributes). Whatever I do I get a On the caller (my laptop):
On both sensors (same error each time, no matter what kind of query type I use):
This seems fishy: |
Sidenote: This error message probably needs some "love":
As in: If neither config nor command line parameters are specified complain about that specifically (and I assume the |
@fako1024 : this thing is going places. You can test with latest changes and should be able to get a result. Will still need polishing, as well as the introduction of the |
Alright, feedback from the next testing round, we're getting a lot closer:
If I limit the query to one of both hosts I get a subset of the results, respectively, so it clearly fetches data and the errors are a lie.
|
@fako1024 : next iteration ready. Have fun 😎
Will revise the request logging when switching API to gin. Agree with your observation. |
Next test looks awesome alrady:
Only found that the Kudos, this looks like an awesome first shot. Can't wait to take a look at the API for integration into other frameworks... 😍 !!! Only remark: As for the HTTP client interaction here I can recommend a cool & well maintained package from some dude (nudge) that simplifies handling of the HTTP client and makes the code more readable (and might also simplify extending the client functionality later on because it already comes with a lot of features, even production-level tested client certificate handling and the likes). Just saying 😉 This is it: https://github.com/fako1024/httpc |
Thanks for the feedback. Glad to see the results looking consistent. As for the Some other news:
It doesn't end here. I'll move the single call part of |
Nice! Moving the query stuff to On a different note though: With the recent changes of yesterday / today the query tool stopped working (independent of the server mode). Doing the same queries as yesterday I now get nothing (literally, no error, no feedback, just a return code of
Diving a little deeper I figured as much: In
Throwing the error at that place at least tells me what's going on:
Could it be that the Just running in server mode seems to work:
|
Also maybe a more fundamental question: What exactly is it that I can |
Interesting. Thanks for the feedback. Then, the latest commit messed up something. The POST should get the query args and then return the result. I'll look at it in the context of moving all the code in entrypoint to the appropriate routine in goQuery (which does the call using the global-query client). Stay tuned |
Thanks for all the testing so far!!! As for proper feedback/error messages: that still needs a bit of love on the client side (I may need your help with regards to how The recent commit can somewhat be called MVP for the entire query system. It should work on your machine. Invocation via
or
The query server is started with
Naturally, this system needs quite some testing in #74 to make sure there are no known regressions/edge cases that aren't covered. @fako1024 : if you can confirm that it's now working in your infrastructure, I'll open the PR. |
Alrighty, got it to run. Some feedback:
I'm unsure if it's only that parameter but since I doubt that you manually re-coded every parameter chances are that more parameters might be affected that are not forwarded to the sensor host
I think the reason is that you did specify a function, but no the intervals. I'll quickly check if that's something that needs handling in One more, unrelated thing: Is the Go API for the server abstract enough already so that it can be integrated into other tools (use case: I happen to have a central control server already and don't want to deploy another binary / independent microservice - instead I just want to integrate the global query API into my existing tool)? |
Thx! Will have a look at the open points. As for your question if the Go API is abstract enough: I would say so - as long as you use |
That would of course be an interesting addition, right. But I was more referring to a "layer below". Let's assume the following scenario: I detect an interesting IP somewhere and now want to figure out if that IP appeared somewhere globally. I should be able to programmatically construct the query.Args in my Go code and then perform a query just like the server / goQuery does (after all, that's how you integrated it into goQuery). Question is only: How much work do I need to do? |
Answer would be: not much work 😄 . As you said, you can create the query args programmatically and then run it using the global-query client's I'm assuming, the whole thing would use JSON under the hood, since you are not planning to display text to a user, but further process the information retrieved. If you do want to display it to a user, take note of the following code inside
That's just adding the label |
Distributed querying for goQuery, aggregating
results.Result
structures. This allows to run queries and flow aggregations against a global fleet which has goProbe/goQuery deployed.The text was updated successfully, but these errors were encountered: