[vtadmin-api] Add /vtexplain endpoint #7528

doeg · 2021-02-21T19:27:20Z

Signed-off-by: Sara Bee 855595+doeg@users.noreply.github.com

Description

This adds the simplest possible /api/vtexplain endpoint to vtadmin-api.

I'm still new to golang, so I invite you to be nit-picky. 😎 It's useful.

Most features I've left for later, like returning JSON instead of text, supporting more (ok... any) of the vtexplain options, and so on. I think it's worth building out more of the front-end first.

Later on, I'd also like to refine some of the errors so they're parseable enough for the front-end to show (ideally) useful feedback. For example, if a serving replica in the keyspace can't be found, "no such tablet" is an accurate error message but not a particularly useful or actionable one for an end-user. I realize I could put more information in the error string, but ultimately I'd like something more robust, like an error envelope with a code (similar to what we did for our internal HTTP handlers). Anyway... Much to consider.

Staged at http://vtadmin.puppy.software:14200/api/vtexplain
Example: http://vtadmin.puppy.software:14200/api/vtexplain?cluster=id1&keyspace=commerce&sql=select%20*%20from%20product

curl "http://vtadmin.puppy.software:14200/api/vtexplain?cluster=id1&keyspace=commerce&sql=select%20*%20from%20product" | jq .

{
  "result": {
    "response": "----------------------------------------------------------------------\nselect * from product\n\n1 commerce/0: select * from product limit 10001\n\n----------------------------------------------------------------------\n"
  },
  "ok": true
}

Related Issue(s)

N/A

Checklist

Should this PR be backported? No
Tests were added or are not required
Documentation was added or is not required

Deployment Notes

N/A

Impacted Areas in Vitess

Components that this PR will affect:

go/vt/vtadmin/api.go

doeg · 2021-02-21T19:37:42Z

go/vt/vtadmin/api.go

+	span, ctx := trace.NewSpan(ctx, "API.VTExplain")
+	defer span.Finish()
+
+	if req.Cluster == "" {


This parameter validation feels a bit clumsy so I'm open to suggestions. Since, in the HTTP handler, these parameters are GET params (i..e, /api/vtexplain?cluster=&keyspace=&sql=) there's no guarantee they'll be defined.

Also, the VTExplain function feels... rather lengthy in general. I'm open to suggestions there too. (That said: even though it's long, it's pretty straightforward, so maybe it's fine...?)

I briefly considered breaking the goroutines out into separate functions, but frankly it made it harder to read + I don't expect the functions have any reusability, since VTExplain is the only instance requiring all this string marshalling.

This parameter validation feels a bit clumsy so I'm open to suggestions. Since, in the HTTP handler, these parameters are GET params (i..e, /api/vtexplain?cluster=&keyspace=&sql=) there's no guarantee they'll be defined.

Also, the VTExplain function feels... rather lengthy in general. I'm open to suggestions there too. (That said: even though it's long, it's pretty straightforward, so maybe it's fine...?)

It would be really cool if there was a good way to split out a mux route into ${parameter validation} + ${logic} phases, where, where the params extracted/parsed/validated by phase 1 are available in the context (not necessarily the literal Go context, but also not necessarily not the literal Go context) of phase 2, but I don't see that, or something like it, from a quick skim of the mux docs. Maybe we should keep thinking about this!

I briefly considered breaking the goroutines out into separate functions, but frankly it made it harder to read + I don't expect the functions have any reusability, since VTExplain is the only instance requiring all this string marshalling.

Agreed that would potentially make things harder to understand, and better to have the longer, but cohesive single function.

VTExplain is the only instance requiring all this string marshalling.

we can also ....................... add a more typed interface to vtexplain :llama_hands:

It would be really cool if there was a good way to split out a mux route into ${parameter validation} + ${logic} phases

Yeah, something along these lines would be useful. We should, at some point, revisit the HTTP response envelopes (particularly for errors) that we're using internally. I think there are ways we can improve on what we have. Or.. we could try gRPC web again, lol.

we can also ....................... add a more typed interface to vtexplain :llama_hands:

I really want this. The string marshalling is easy to mess up + is a bit tedious to debug.

jmoldow · 2021-02-21T20:12:32Z

Cool, glad to see this! In case it helps, I wrote down some ideas in #7333 about how arguments could potentially be supported for an integrated vtexplain.

doeg · 2021-02-21T20:22:44Z

@jmoldow wow, this is great! Smart defaults! ✨ I think this would be a really fantastic addition. Big thanks for sharing that. I'll start thinking about this from the front-end perspective.

I can definitely loop you in on further VTExplain-in-VTAdmin work too, if you're interested! And if you have any ideas or thoughts on UI, I'd particularly love to hear 'em. 😎

deepthi · 2021-02-22T16:25:46Z

Re error messages, #7514 might help once it is completed.

ajm188

This looks great! Everything I have is just extra suggestions and nitpicks, but lg2m to merge as-is

go/vt/vtadmin/api.go

ajm188 · 2021-02-22T20:42:12Z

go/vt/vtadmin/api.go

@@ -534,6 +540,47 @@ func (api *API) getTablets(ctx context.Context, c *cluster.Cluster) ([]*vtadminp
 	return ParseTablets(rows, c)
 }

+// findTablet returns the first tablet in a given cluster that satisfies the filter function.


Maybe these functions make more sense on the cluster.Cluster type? That would at least help break this file up a bit. Totally up to you!

I would definitely like to move them elsewhere. I think, though, that moving them to cluster.Cluster might introduce a weird dependency.

Specifically, per your suggestion, I want to move the following three functions out of vtadmin/api.go and into vtadmin/cluster/cluster.go:

findTablet

findTablets

getTablets, which is called by the two above

However, getTablets calls ParseTablets, which is in vtadmin/tablets.go (package vtadmin), which would (I think) cause a circular dependency. I assume this is undesirable in golang, right?

I have two ideas, not sure which (if either) make sense to do:

Create a separate package tablet vtadmin/tablet/tablet.go (analogous to vtadmin/cluster/cluster.go)

Move the three functions listed above into vtadmin/tablets.go, since the other functions in tablets.go also take a cluster argument. It seems like not a bad fit.

Happy to take any suggestion you have!

Yeah, I think anything that is like func (api *API) DoThingWithTablets(ctx context.Context, c *cluster.Cluster) we should try (it might have some other dependencies on api, which could get uhhhh interesting) to make it a func (c *Cluster) DoThingWithTablets(ctx context.Context)

@ajm188 hmmm ok. In that case, I will have to move tablet.go (and its ParseTablet function) into its own package tablet so cluster.go can use it without adding a (circular) dependency on package vtadmin, right? It seems reasonable to me, I just want to double check before I move things around.

Yeah, that's what I was saying with where it should be func (c *Cluster) ParseTablets(ctx context.Context, ...) ([]*vtadminpb.Tablet, error) instead of func (api *API) ParseTablets(ctx context.Context, c *cluster.Cluster, ...) (...).

Also, I'm not super convinced that needs to remain a public function. It doesn't really feel like part of the API of a cluster, while "find N tablets in the cluster" definitely does.

@ajm188 that sounds really good.

Moving things around ends up being a rather big + messy commit, and it'd be nice to merge this one in since I'm branched off of it for front-end reasons.

Would it be ok with you if I moved everything into clusters.go in a separate PR? I can do it today, right after this, and add a bunch of tests as penance.

Yeah, let's file an issue for that, because I'll forget, but sg2m to merge

ajm188 · 2021-02-22T20:54:19Z

go/vt/vtadmin/api.go

+	span, ctx := trace.NewSpan(ctx, "API.VTExplain")
+	defer span.Finish()
+
+	if req.Cluster == "" {


This parameter validation feels a bit clumsy so I'm open to suggestions. Since, in the HTTP handler, these parameters are GET params (i..e, /api/vtexplain?cluster=&keyspace=&sql=) there's no guarantee they'll be defined.

Also, the VTExplain function feels... rather lengthy in general. I'm open to suggestions there too. (That said: even though it's long, it's pretty straightforward, so maybe it's fine...?)

It would be really cool if there was a good way to split out a mux route into ${parameter validation} + ${logic} phases, where, where the params extracted/parsed/validated by phase 1 are available in the context (not necessarily the literal Go context, but also not necessarily not the literal Go context) of phase 2, but I don't see that, or something like it, from a quick skim of the mux docs. Maybe we should keep thinking about this!

I briefly considered breaking the goroutines out into separate functions, but frankly it made it harder to read + I don't expect the functions have any reusability, since VTExplain is the only instance requiring all this string marshalling.

Agreed that would potentially make things harder to understand, and better to have the longer, but cohesive single function.

VTExplain is the only instance requiring all this string marshalling.

we can also ....................... add a more typed interface to vtexplain :llama_hands:

go/vt/vtadmin/api.go

Signed-off-by: Sara Bee <855595+doeg@users.noreply.github.com>

deepthi · 2021-02-23T20:56:22Z

This is so awesome as a feature, I can't even!

doeg force-pushed the sarabee-experiments-in-vtexplain branch from d362b20 to a938277 Compare February 21, 2021 19:27

doeg commented Feb 21, 2021

View reviewed changes

go/vt/vtadmin/api.go Outdated Show resolved Hide resolved

doeg commented Feb 21, 2021

View reviewed changes

doeg marked this pull request as ready for review February 21, 2021 19:41

doeg requested a review from ajm188 as a code owner February 21, 2021 19:41

doeg requested review from deepthi, rohit-nayak-ps and systay February 21, 2021 19:42

doeg changed the title ~~Add /vtexplain endpoint~~ [vtadmin-api] Add /vtexplain endpoint Feb 21, 2021

jmoldow mentioned this pull request Feb 21, 2021

Feature Request: Integrate VTExplain into vtadmin/vtctld/vtctl/vtctlclient, vtgate/vtclient #7333

Open

ajm188 approved these changes Feb 22, 2021

View reviewed changes

doeg added 3 commits February 22, 2021 21:50

Add /vtexplain endpoint

a2f4650

Signed-off-by: Sara Bee <855595+doeg@users.noreply.github.com>

Update VTExplain to use IsInServingGraph for findTablet lookup

1a8e556

Signed-off-by: Sara Bee <855595+doeg@users.noreply.github.com>

🐐 topodata -> topodatapb

3a2b95e

Signed-off-by: Sara Bee <855595+doeg@users.noreply.github.com>

doeg force-pushed the sarabee-experiments-in-vtexplain branch from 3dfc53a to 3a2b95e Compare February 23, 2021 02:51

Update TestVTExplain to use testutil.TabletManagerClient

e9508d7

Signed-off-by: Sara Bee <855595+doeg@users.noreply.github.com>

ajm188 approved these changes Feb 23, 2021

View reviewed changes

deepthi merged commit 81e6480 into vitessio:master Feb 23, 2021

doeg mentioned this pull request Feb 25, 2021

[vtadmin-api] Reorganize tablet-related functions into vtadmin/cluster/cluster.go #7553

Merged

8 tasks

askdba added the Component: VTAdmin VTadmin interface label Feb 26, 2021

askdba added this to the v10.0 milestone Feb 26, 2021

doeg added this to In progress in VTAdmin via automation Mar 16, 2021

doeg moved this from In progress to Done in VTAdmin Mar 16, 2021

ajm188 mentioned this pull request Jul 14, 2021

slack vitess v10.pre tinyspeck/vitess#228

Merged

doeg deleted the sarabee-experiments-in-vtexplain branch November 1, 2021 19:46

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[vtadmin-api] Add /vtexplain endpoint #7528

[vtadmin-api] Add /vtexplain endpoint #7528

doeg commented Feb 21, 2021 •

edited

doeg Feb 21, 2021 •

edited

ajm188 Feb 22, 2021

doeg Feb 23, 2021

jmoldow commented Feb 21, 2021

doeg commented Feb 21, 2021

deepthi commented Feb 22, 2021

ajm188 left a comment

ajm188 Feb 22, 2021

doeg Feb 23, 2021 •

edited

ajm188 Feb 23, 2021 •

edited

doeg Feb 23, 2021 •

edited

ajm188 Feb 23, 2021

doeg Feb 23, 2021

ajm188 Feb 23, 2021

ajm188 Feb 22, 2021

deepthi commented Feb 23, 2021

[vtadmin-api] Add /vtexplain endpoint #7528

[vtadmin-api] Add /vtexplain endpoint #7528

Conversation

doeg commented Feb 21, 2021 • edited

Description

Related Issue(s)

Checklist

Deployment Notes

Impacted Areas in Vitess

doeg Feb 21, 2021 • edited

Choose a reason for hiding this comment

ajm188 Feb 22, 2021

Choose a reason for hiding this comment

doeg Feb 23, 2021

Choose a reason for hiding this comment

jmoldow commented Feb 21, 2021

doeg commented Feb 21, 2021

deepthi commented Feb 22, 2021

ajm188 left a comment

Choose a reason for hiding this comment

ajm188 Feb 22, 2021

Choose a reason for hiding this comment

doeg Feb 23, 2021 • edited

Choose a reason for hiding this comment

ajm188 Feb 23, 2021 • edited

Choose a reason for hiding this comment

doeg Feb 23, 2021 • edited

Choose a reason for hiding this comment

ajm188 Feb 23, 2021

Choose a reason for hiding this comment

doeg Feb 23, 2021

Choose a reason for hiding this comment

ajm188 Feb 23, 2021

Choose a reason for hiding this comment

ajm188 Feb 22, 2021

Choose a reason for hiding this comment

deepthi commented Feb 23, 2021

doeg commented Feb 21, 2021 •

edited

doeg Feb 21, 2021 •

edited

doeg Feb 23, 2021 •

edited

ajm188 Feb 23, 2021 •

edited

doeg Feb 23, 2021 •

edited