[feature] Distributed query infrastructure (global-query server) #109

els0r · 2023-04-11T20:41:47Z

Time to let the cat out of the bag. This is the first working version with

distributed query capabilities
client/server logging
a unified query client for goProbe and global-query
massive code simplifcation and re-use across the entire query pipeline

It also moves the entire goProbe API into legacyapi, not serving it anymore. This is preparation for #58, which should then be a more concise PR than this beast.

Closes #43

fako1024

Obviously no guarantees to safety of the review (my head's spinning after going through this behemoth of a PR 💫 ).
Change requested because of the raw query issue (not sure if others are affected, too). I'll try to help investigate as much as I can.

fako1024 · 2023-04-12T07:49:08Z

cmd/global-query/api-client-querier.yaml

Should this maybe go someplace else (e.g. addon or examples or similar)?

I like to keep it next to the binary, in case someone wants to run the tool directly inside the directory with go run main.go --config .....

But no strong opinion here.

cmd/global-query/cmd/root.go

cmd/global-query/cmd/server.go

fako1024 · 2023-04-12T07:58:30Z

cmd/global-query/local-config.yaml

Should this maybe go someplace else (e.g. addon or examples or similar)?

We could. Then we should supply them for every binary though.

cmd/global-query/pkg/conf/conf.go

pkg/api/goprobe/client/client.go

pkg/api/middleware.go

pkg/api/query.go

pkg/results/result.go

pkg/types/columns.go

els0r · 2023-04-12T11:51:28Z

Obviously no guarantees to safety of the review (my head's spinning after going through this behemoth of a PR 💫 ).

Change requested because of the raw query issue (not sure if others are affected, too). I'll try to help investigate as much as I can.

Thanks for the initial review and the suggestions.

I'll also prioritize looking at the raw queries. Unfortunately never ran them during testing 🙈.

els0r

@fako1024 : I've violently broken the build by rebasing on develop. Now it's complaining about changes that come from slimcap. Could you help me with this? I don't want to fix something that ain't broken.

pkg/types/columns.go

pkg/results/result.go

cmd/global-query/pkg/conf/conf.go

els0r · 2023-04-13T13:23:15Z

pkg/api/client/client.go

+	// DO NOT uncomment before clarifying behavior with httpc library
+	//
+	// retry any request that isn't 2xx
+	// req = req.RetryBackOffErrFn(func(resp *http.Response, _ error) bool {


No problem. I really appreciate the conciseness of the library. At least 10x better than all the boilerplate repetition to make http requests.

Your suggestion with the interval generators sounds really good and simple to use. Then you don't have to modify the RetryBackOff function signature and just provide some syntactic sugar on top. Nice!

fako1024 · 2023-04-14T09:47:36Z

@fako1024 : I've violently broken the build by rebasing on develop. Now it's complaining about changes that come from slimcap. Could you help me with this? I don't want to fix something that ain't broken.

Sure thing, I got your back! I've reverted slimcap to the latest version without breaking API changes. I'll have to make some changes in goProbe before bumping it to the newest commit. Build is fine again: https://github.com/els0r/goProbe/actions/runs/4698522445/jobs/8330904665

Gets rid of the single invocation use for global-query and now requires the tool to run in server mode. Various other fixes and adjustments and preparation for a single api folder that stores all API clients and servers for the involved tools

Also adds a basic scaffold for setting tracing information (tbc).

… clients Resides somewhere in the RetryBackoff function of the httpc library.

els0r · 2023-05-01T02:23:32Z

@fako1024 : given that the review feedback has been incorporated and the major blocker (time queries not working) has been resolved, may I have your blessings?

The ideas are storming into my head and I'd like to tick them off in more digestable chunks and in separate issues. The API restructure of goProbe, for example, is next.

Toodles and hi to the old man.

fako1024 · 2023-05-01T14:10:48Z

@fako1024 : given that the review feedback has been incorporated and the major blocker (time queries not working) has been resolved, may I have your blessings?

The ideas are storming into my head and I'd like to tick them off in more digestable chunks and in separate issues. The API restructure of goProbe, for example, is next.

Toodles and hi to the old man.

Sure thing, let's go ahead. Thanks for following up on the feedback. Side note: I've recently released httpc, incorporating all the recent changes / fixes you triggered: https://github.com/fako1024/httpc/releases/tag/v1.0.14 - Feel free to bump the current version in go.mod to this one if you'd like.

els0r · 2023-05-03T00:00:46Z

@fako1024 : given that the review feedback has been incorporated and the major blocker (time queries not working) has been resolved, may I have your blessings?

The ideas are storming into my head and I'd like to tick them off in more digestable chunks and in separate issues. The API restructure of goProbe, for example, is next.

Toodles and hi to the old man.

Sure thing, let's go ahead. Thanks for following up on the feedback. Side note: I've recently released httpc, incorporating all the recent changes / fixes you triggered: https://github.com/fako1024/httpc/releases/tag/v1.0.14 - Feel free to bump the current version in go.mod to this one if you'd like.

Many thanks. Will do another bump in go.mod of httpc when reworking the API and clients.

els0r linked an issue Apr 11, 2023 that may be closed by this pull request

global-query #43

Closed

els0r changed the title ~~43 global query~~ [feature] Distributed query infrastructure (global-query server) Apr 11, 2023

els0r requested a review from fako1024 April 11, 2023 20:42

fako1024 mentioned this pull request Apr 12, 2023

[bugfix] Straighten out command-line error handling #106

Merged

2 tasks

fako1024 requested changes Apr 12, 2023

View reviewed changes

els0r force-pushed the 43-global-query branch 2 times, most recently from 82c7a65 to e4f95f4 Compare April 13, 2023 03:45

els0r commented Apr 13, 2023

View reviewed changes

Lennart Elsen added 20 commits April 30, 2023 20:41

Proof of concept implementation for global-query

6896174

Fix status and error handling

a70b028

Simplifcation of the Runner interface

19b4b51

Change the Runner interface to accept query.Args

d13aa68

Fix issues with results aggregation and summary computation

9256754

Fix version panic in goProbe

7964bca

Adds support for printing hostname and hostid

f8897e0

Make sure a host identifier is appended to a global query

8cefeca

Merging of metadata

f448c3b

Proper separation of query setup and distributed query

e28c9d9

Use httpc for client construction

a6bf2e3

Distributed query server support

4acb1f8

Client for the distributed query server

f651dec

Add DB path to goquery config

efb628b

Shared client and server stubs for API

63fc14f

Fix discovery test

9f6d006

Fix config tests

ea59f75

Add logging middleware

9a69711

Structured request logging for clients and servers

8345ea2

Also adds a basic scaffold for setting tracing information (tbc).

Lennart Elsen and others added 5 commits April 30, 2023 20:44

Fix bug where results aren't deserialized when using one of goprobe's…

61efd88

… clients Resides somewhere in the RetryBackoff function of the httpc library.

Fix totals calculation and propagate more concise errors

e21017e

Rebase and dependency update

2d4b17e

Revert depdencency update to avoid API breaking

c2dc228

Rebase off develop

79cd314

els0r force-pushed the 43-global-query branch from 02c0423 to 79cd314 Compare May 1, 2023 00:45

Lennart Elsen added 3 commits April 30, 2023 20:55

Follow cmd naming scheme for goQuery

fbffc09

Namespace long opts of command line parameters

d9a0e9e

Add simple retry behavior back to the client

e750163

els0r requested a review from fako1024 May 1, 2023 02:23

fako1024 approved these changes May 1, 2023

View reviewed changes

els0r merged commit 2898236 into develop May 3, 2023

els0r deleted the 43-global-query branch May 3, 2023 00:01

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[feature] Distributed query infrastructure (global-query server) #109

[feature] Distributed query infrastructure (global-query server) #109

els0r commented Apr 11, 2023 •

edited

Loading

fako1024 left a comment

fako1024 Apr 12, 2023

els0r May 1, 2023

fako1024 Apr 12, 2023

els0r Apr 13, 2023

els0r commented Apr 12, 2023

els0r left a comment

els0r Apr 13, 2023

fako1024 commented Apr 14, 2023

els0r commented May 1, 2023

fako1024 commented May 1, 2023

els0r commented May 3, 2023

[feature] Distributed query infrastructure (global-query server) #109

[feature] Distributed query infrastructure (global-query server) #109

Conversation

els0r commented Apr 11, 2023 • edited Loading

fako1024 left a comment

Choose a reason for hiding this comment

fako1024 Apr 12, 2023

Choose a reason for hiding this comment

els0r May 1, 2023

Choose a reason for hiding this comment

fako1024 Apr 12, 2023

Choose a reason for hiding this comment

els0r Apr 13, 2023

Choose a reason for hiding this comment

els0r commented Apr 12, 2023

els0r left a comment

Choose a reason for hiding this comment

els0r Apr 13, 2023

Choose a reason for hiding this comment

fako1024 commented Apr 14, 2023

els0r commented May 1, 2023

fako1024 commented May 1, 2023

els0r commented May 3, 2023

els0r commented Apr 11, 2023 •

edited

Loading