[vtadmin] Add vtctld proxy to vtadmin API, add GetKeyspaces endpoint #7266

ajm188 · 2021-01-08T00:55:26Z

~~(I still need to deploy this fully to test, hence draft. But all feedback welcome!!)~~ Confirmed it works, yay!

Description

This does the following:

Adds vtctldclient.Proxy to vtadmin, semantically similar to vtsql.DB but for vtctldclient connections.
Adds GetKeyspaces rpc to the vtadmin service, and implements it
(minor) testing refactors for grpcvtctldserver, grpcvtctldclient
(minor) adds vtctld discovery to the discovery interface, implements for all current implementations (static + consul)

Related Issue(s)

Checklist

Should this PR be backported? No
Tests were added or are not required Yes
Documentation was added or is not required Yes

Deployment Notes

Impacted Areas in Vitess

Components that this PR will affect:

Query Serving
VReplication
Cluster Management (I'm going to add a category for vtadmin like I promised!!!)
Build

Signed-off-by: Andrew Mason <amason@slack-corp.com>

This will allow both vtsql and vtctldclient to access it Signed-off-by: Andrew Mason <amason@slack-corp.com>

Signed-off-by: Andrew Mason <amason@slack-corp.com>

…, and vtadmin Signed-off-by: Andrew Mason <amason@slack-corp.com>

Signed-off-by: Andrew Mason <amason@slack-corp.com>

ajm188 · 2021-01-08T00:57:32Z

go/vt/vtadmin/api_test.go

 	"vitess.io/vitess/go/vt/vitessdriver"
 	"vitess.io/vitess/go/vt/vtadmin/cluster"
 	"vitess.io/vitess/go/vt/vtadmin/cluster/discovery/fakediscovery"
 	"vitess.io/vitess/go/vt/vtadmin/grpcserver"
 	"vitess.io/vitess/go/vt/vtadmin/http"
+	vtadminvtctldclient "vitess.io/vitess/go/vt/vtadmin/vtctldclient"


This is, ahem, admittedly extremely gross. Please help lol

We could call it vtctldclient given that we're already in a "vtadmin" context. 🧐 Up to you though, I don't think this is any worse than all the other vtctld-related consonant salads lol.

It conflicts with the import on line 40

w e l p :')

Signed-off-by: Andrew Mason <amason@slack-corp.com>

doeg · 2021-01-09T17:52:32Z

(I still need to deploy this fully to test, hence draft. But all feedback welcome!!)

This PR is still in progress right? 🐈 No rush obviously, just wanna make sure you're not waiting on me for reviews! :') (It looks great so far, but what else is new!)

ajm188 · 2021-01-09T18:37:49Z

This PR is still in progress right? 🐈 No rush obviously, just wanna make sure you're not waiting on me for reviews! :') (It looks great so far, but what else is new!)

It's "in-progress" only in the sense that I may need to make some minor tweaks to iron out any bugs after deploying (for ex yesterday I found "oh right, I never made this reachable over HTTP"). But! I feel good enough about the rest of the implementation (basically everything below the layers in api.go.

doeg

This looks great! (And... familiar! hehehe). I encourage you to get another golang +1, though, since this PR is so big. :)

Tomorrow I'll run this locally using static file discovery (so we can say we can say we did, and because I'll have to for vtadmin-web anyway). I'll also update the vtadmin-web README until we get a better local dev env set up. :') After that, I'll officially approve this bad boy. ✨

Thank you so much for doing this! I'm so excited for our upcoming vrep endpoints!!!

doeg · 2021-01-13T21:25:25Z

go/vt/vtadmin/cluster/discovery/discovery.go

+	// further. Which vtctld in a set of found vtctlds is returned is not
+	// specified by the interface, and can be implementation-specific.
+	DiscoverVtctld(ctx context.Context, tags []string) (*vtadminpb.Vtctld, error)
+	// DiscoverVtctldAddr returns the address of a of vtctld found in the


Typo: "of a of"

doeg · 2021-01-13T21:25:44Z

go/vt/vtadmin/cluster/discovery/discovery.go

+	DiscoverVtctld(ctx context.Context, tags []string) (*vtadminpb.Vtctld, error)
+	// DiscoverVtctldAddr returns the address of a of vtctld found in the
+	// discovery service. Tags can optionally be used to filter the set of
+	// potential vtctld further. Which gate in a set of found vtctld is used to


Typo: "potential vtctlds" + "found vtctlds"

And also "which gate" whoops!

doeg · 2021-01-13T21:28:27Z

go/vt/vtadmin/cluster/discovery/discovery_static_file.go

@@ -52,11 +52,16 @@ type StaticFileDiscovery struct {
 		byName map[string]*vtadminpb.VTGate
 		byTag  map[string][]*vtadminpb.VTGate
 	}
+	vtctlds struct {


Thanks for making the changes to this file (even though I guess it was uh, mandatory) 🙏

doeg · 2021-01-13T21:38:12Z

go/vt/vtadmin/http/keyspaces.go

+)
+
+// GetKeyspaces implements the http wrapper for /keyspaces[?cluster=[&cluster=]].
+func GetKeyspaces(ctx context.Context, r Request, api *API) *JSONResponse {


Signed-off-by: Andrew Mason <amason@slack-corp.com>

rohit-nayak-ps

lgtm

doeg · 2021-01-14T21:03:23Z

go/vt/vtadmin/cluster/discovery/discovery_static_file.go

+
+	count := len(vtctlds)
+	if count == 0 {
+		return nil, ErrNoVTGates


Sorry for missing this before -- I think this should be ErrNoVtctlds.

Fixed! Nice catch!

doeg

Apart from this tiny comment, this (still) looks great. ✨

I was able to get it almost but not entirely working against local Docker/Vitess + vtadmin-web... I think I'm missing a couple of flags. 🤔 Given that you deployed this internally + it is working great, I don't think we should block this merge on me. We can figure it out when you're back. 😊

ajm188 · 2021-01-15T12:11:09Z

Happy to help with flags today (or later)!

Signed-off-by: Andrew Mason <amason@slack-corp.com>

[vtadmin] Add vtctld proxy to vtadmin API, add GetKeyspaces endpoint

ajm188 added 12 commits January 7, 2021 19:28

Add a proto that requires vtctld access

be4d68a

Signed-off-by: Andrew Mason <amason@slack-corp.com>

Start implementing vtctld discovery

eecbf8c

Signed-off-by: Andrew Mason <amason@slack-corp.com>

Extract the datacenter template generation

b9198f4

Signed-off-by: Andrew Mason <amason@slack-corp.com>

Move the LoadFromTemplate helper to a separate package

c8d506d

This will allow both vtsql and vtctldclient to access it Signed-off-by: Andrew Mason <amason@slack-corp.com>

Add grpcvtctldclient constructor for use in vtadmin

3370c06

Signed-off-by: Andrew Mason <amason@slack-corp.com>

Add minimally-viable vtctldclient Proxy package

939aff6

Signed-off-by: Andrew Mason <amason@slack-corp.com>

Add vtctld config support to cluster, update tests

e2cc785

Signed-off-by: Andrew Mason <amason@slack-corp.com>

Add testutil for grpcvtctldserver, to use both here, grpcvtctldclient…

b005279

…, and vtadmin Signed-off-by: Andrew Mason <amason@slack-corp.com>

Adapt buildCluster to also inject a vtctld client

2d3d6f8

Signed-off-by: Andrew Mason <amason@slack-corp.com>

Add GetKeyspaces implementation and tests

5091f7c

Signed-off-by: Andrew Mason <amason@slack-corp.com>

Implement vtctld discovery for staticfile

e4e37a9

Signed-off-by: Andrew Mason <amason@slack-corp.com>

Add staticfile tests for vtctld discovery

dc6ebc8

Signed-off-by: Andrew Mason <amason@slack-corp.com>

ajm188 requested review from doeg and setassociative January 8, 2021 00:55

ajm188 commented Jan 8, 2021

View reviewed changes

ajm188 force-pushed the am_vtadmin_vtctld branch from c4d91c5 to d6faf28 Compare January 8, 2021 02:07

Fix lints

2a032aa

Signed-off-by: Andrew Mason <amason@slack-corp.com>

ajm188 force-pushed the am_vtadmin_vtctld branch from d6faf28 to 2a032aa Compare January 8, 2021 02:10

ajm188 added 2 commits January 8, 2021 08:28

Add HTTP route

9b926a3

Signed-off-by: Andrew Mason <amason@slack-corp.com>

Need to pair WithInsecure with WithPerRPCCredentials

eea6114

Signed-off-by: Andrew Mason <amason@slack-corp.com>

ajm188 requested review from rohit-nayak-ps and deepthi January 13, 2021 20:51

ajm188 marked this pull request as ready for review January 13, 2021 20:51

ajm188 requested a review from sougou as a code owner January 13, 2021 20:51

doeg reviewed Jan 13, 2021

View reviewed changes

Fix mistyped documentation

01ea357

Signed-off-by: Andrew Mason <amason@slack-corp.com>

rohit-nayak-ps approved these changes Jan 14, 2021

View reviewed changes

doeg reviewed Jan 14, 2021

View reviewed changes

doeg approved these changes Jan 14, 2021

View reviewed changes

Fix error name from copy-paste :shamebell:

00fba1e

Signed-off-by: Andrew Mason <amason@slack-corp.com>

rohit-nayak-ps merged commit 36110d8 into vitessio:master Jan 15, 2021

doeg mentioned this pull request Jan 15, 2021

Expose vtctld gRPC port in local Docker example + update VTAdmin README #7306

Merged

8 tasks

askdba added this to the v9.0 milestone Jan 19, 2021

doeg added this to In progress in VTAdmin via automation Mar 16, 2021

doeg moved this from In progress to Done in VTAdmin Mar 16, 2021

rafael pushed a commit to tinyspeck/vitess that referenced this pull request Apr 5, 2021

Merge pull request vitessio#7266 from tinyspeck/am_vtadmin_vtctld

161bac8

[vtadmin] Add vtctld proxy to vtadmin API, add GetKeyspaces endpoint

ajm188 added Component: VTAdmin VTadmin interface Type: Feature Request labels May 21, 2021

ajm188 deleted the am_vtadmin_vtctld branch May 29, 2021 10:25

eseokoh mentioned this pull request May 31, 2021

Release Note of v9.0.0 includes changes of v10.0 #8212

Closed

ajm188 mentioned this pull request Jul 12, 2021

slack vitess v10.pre tinyspeck/vitess#228

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[vtadmin] Add vtctld proxy to vtadmin API, add GetKeyspaces endpoint #7266

[vtadmin] Add vtctld proxy to vtadmin API, add GetKeyspaces endpoint #7266

ajm188 commented Jan 8, 2021 •

edited

ajm188 Jan 8, 2021

doeg Jan 9, 2021

ajm188 Jan 9, 2021

doeg Jan 9, 2021

doeg commented Jan 9, 2021

ajm188 commented Jan 9, 2021

doeg left a comment

doeg Jan 13, 2021

doeg Jan 13, 2021

ajm188 Jan 13, 2021

doeg Jan 13, 2021

doeg Jan 13, 2021

rohit-nayak-ps left a comment

doeg Jan 14, 2021

ajm188 Jan 15, 2021

doeg left a comment •

edited

ajm188 commented Jan 15, 2021

[vtadmin] Add vtctld proxy to vtadmin API, add GetKeyspaces endpoint #7266

[vtadmin] Add vtctld proxy to vtadmin API, add GetKeyspaces endpoint #7266

Conversation

ajm188 commented Jan 8, 2021 • edited

Description

Related Issue(s)

Checklist

Deployment Notes

Impacted Areas in Vitess

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

doeg commented Jan 9, 2021

ajm188 commented Jan 9, 2021

doeg left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rohit-nayak-ps left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

doeg left a comment • edited

Choose a reason for hiding this comment

ajm188 commented Jan 15, 2021

ajm188 commented Jan 8, 2021 •

edited

doeg left a comment •

edited