feat(discovery): pluggable discovery services API #1028
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
4 tasks
andrewazores
force-pushed
the
939-story-pluggable-discovery-api
branch
4 times, most recently
from
ee5f7ca
to
a4799d8
Compare
This was referenced Aug 3, 2022
andrewazores
force-pushed
the
939-story-pluggable-discovery-api
branch
3 times, most recently
from
d4feb8a
to
168cd9f
Compare
andrewazores
force-pushed
the
939-story-pluggable-discovery-api
branch
from
168cd9f
to
862f627
Compare
andrewazores
force-pushed
the
939-story-pluggable-discovery-api
branch
2 times, most recently
from
3ff6233
to
580d4a6
Compare
andrewazores
force-pushed
the
939-story-pluggable-discovery-api
branch
2 times, most recently
from
80096dc
to
1c995c9
Compare
|
Ready for review. This is just merging this long-lived upstream feature branch into upstream main. The component PRs of the feature have already been reviewed separately and merged into this branch. |
* Initial rough cut Adds API endpoints for pluggable discovery. Also introduces DiscoveryStorage which is either a stand-in for a database ORM or an abstraction around a later database connection to come. BuiltInDiscovery is used as a bridge between existing PlatformClient implementations and platform-specific discovery behaviours and the DiscoveryStorage component. This allows existing PlatformClients to continue performing discovery within the main Cryostat process, but publish those results into the DiscoveryStorage when changes occur. API endpoints that query for a list of targets or the discovery tree read from DiscoveryStorage so that these calls do not cascade out into further network calls (ex. for KubeApiPlatformClient). * DiscoveryStorage implements PlatformClient The new DiscoveryStorage replaces the previous platform detection mechanism and provided PlatformClient, and the MergingPlatformClient. The CustomTargetPlatformClient and the platform-detected appropriate PlatformClient implementation are injected into a `Set<PlatformClient>` which the BuiltInDiscovery consumes and uses to bridge updates into the DiscoveryStorage. Since DiscoveryStorage itself implements the PlatformClient interface, but is able to return results from storage rather than re-querying the base platform, it is used as a replacement injection in various existing places that injected the PlatformClient instance previously (which would have always received the MergingPlatformClient instance). * fixup! Initial rough cut * fixup! fixup! Initial rough cut * Refactor and cleanup * add discovery deregistration endpoint * builtin discovery can be disabled * add comment * use UUID string for plugin registration ID, not numeric * include placeholder auth token in registration response * handle null children set as empty set * internally-added callback URIs may be null non-nullness is enforced on API endpoint. this allows internal cases like BuiltInDiscovery to register themselves through the discovery mechanism as an internal API client, without having to expose some meaningless callback URL since the Cryostat process is the "plugin" process * experimental fixes * don't send previous state back to plugin * fix typos * implement gson adapter for parsing subtrees provided by discovery plugins * add quarkus-test sample using pluggable discovery * builtin discovery completes startup promise * discovery storage is also a verticle * user basic auth * fix image tag * emit warnings on unexpected object members clears spotbugs warning for missing 'default' in switch * fixup webPort for quarkus-test-plugin
* feat(discovery): prepare to implement URL callback handling persist to disk on shutdown and restore from disk on startup to emulate database connection persistence. On startup, handle callback URLs restored and check if discovery services are still known/present. * don't persist or restore plugins without callbacks API registration endpoint enforces callback URI presence, so only builtin discovery mechanisms can have null callbacks. These should not be persisted * POST plugin callback URLs on startup to check presence * run discovery plugin with restart policy plugin will exit itself on startup if registration fails * improve shutdown handling if verticles fail to start up * refactor startup for cleanliness * don't register self if builtin discovery disabled * respond 400 if UUID invalid * add handler unit tests, minor fixups * use newer vertx-fib-demo that handles SIGTERM * auto generate basic auth credentials file
* feat(db): add in-memory h2 database w/ Hibernate replace discovery file-based storage with in-memory h2 * extract StorageModule * extract AbstractDao * make database parameters configurable * format sql in logs * use jsonb for subtrees * use file-based h2 by default * minor cleanup * rough setup for postgres db and switching from h2 in-memory, h2 file, and postgres remote * minor postgres fixes * remove thread sleep hack for dependent startup * don't fail startup if a discovery plugin is offline * extract VerticleDeployer * refactor * only update schema in file mode, don't (re)create * handle target discovery events async * emit target discovery ws notifications async * detach entity on save * use correct exception type * Revert "emit target discovery ws notifications async" This reverts commit cd78257. * fixup! extract VerticleDeployer * use normalized form of env var names * avoid addChildNode mutation * cleanup and add tests * update plugin sample app version * fixup! cleanup and add tests * avoid redundant null check * remove useless file * query logging can be disabled * remove json file persistence from custom targets, update unit tests, add itest * fixup! remove json file persistence from custom targets, update unit tests, add itest * builtin discovery revamp on startup, check if a realm is registered for each built-in discovery mechanism (platform client). register a plugin realm if not, then attach an update listener to the platform client to update that realm. delegate to the platform client to perform on-start load of current known state to replace existing persistent storage contents. Custom Targets overrides this and does not replace existing storage contents, so that the discovery database is the backing storage for custom targets * configure postgres container to self-init on startup, no need for separate init command * re-initialize custom targets from database on startup allows custom targets to be deleted on subsequent runs
andrewazores
force-pushed
the
939-story-pluggable-discovery-api
branch
from
1c995c9
to
6b68364
Compare
andrewazores
force-pushed
the
939-story-pluggable-discovery-api
branch
from
c5ca710
to
55b7072
Compare
…unique callback URLs
andrewazores
force-pushed
the
939-story-pluggable-discovery-api
branch
from
55b7072
to
f225aab
Compare
maxcao13
approved these changes
Aug 18, 2022
This was referenced Aug 19, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Closes #939
Closes #938 (the subtree update model and built-in discovery mechanisms already work this way)
Includes: