validate command

[uwedeportivo]

Adds a new validate command to src-cli. This command validates a Sourcegraph instance by executing a script given as an argument. The intention is to provide a way for admins of Sourcegraph instances to verify that their new install or their upgrade went well and the instance is functioning.

The validation steps are specified in a json structure. The validation follows this outline:

• create the first admin user
• use their credentials to create an external service
• wait for repo to clone
• execute a search and check the count on the result
• delete the external service

The spec is something like this (if some of the spec structs are not declared, then the corresponding step is skipped)

{
  "firstAdmin": {
    "email": "e2e@sourcegrah.com",
    "username": "e2e-test-user",
    "password": "123123123-e2e-test"
  },
  "externalService": {
    "config": {
      "url": "https://github.com",
      "token": "{{ .github_token }}",
      "orgs": [],
      "repos": [
        "sourcegraph-testing/zap"
      ]
    },
    "kind": "GITHUB",
    "displayName": "footest",
    "deleteWhenDone": true
  },
  "waitRepoCloned": {
    "repo": "github.com/sourcegraph-testing/zap",
    "maxTries": 5,
    "sleepBetweenTriesSeconds": 2
  },
  "searchQuery": "repo:^github.com/sourcegraph-testing/zap$ SugaredLogger count:99999"
}

documentation for the command is here sourcegraph/sourcegraph#12001

[dadlerj]

For the customer use case, it's nice to have a flexible tool for this, but I worry that it's too complex/time-intensive to expect them to write their own tests. Would we ship this with a built-in test suite that we recommend?

[uwedeportivo]

For the customer use case, it's nice to have a flexible tool for this, but I worry that it's too complex/time-intensive to expect them to write their own tests. Would we ship this with a built-in test suite that we recommend?

We will provide examples that customers can edit and insert their own repo, token, external services details. Does that address your concerns ? Customers don't need to use this if they don't feel a need for it. But I think the simple syntax and the provided docs and examples will make this not be an issue imho.

[unknwon]

This is great stuff! But I have the similar concern as @dadlerj that writing this kind of tests actually requires the customer/admin being proficient of how Sourcegraph actually works internally?

[uwedeportivo]

@unknwon i'm not sure i understand your comment: why would admins need to know the internals. the scripting only exposes operations that one would do in the UI or in the graphql API console.

[dadlerj]

My POV:

As long as (eventually) no customers need to learn this command language, I'm happy. If the user experience is src validate <default-smoke-test-script> that they can run after upgrading, then I'd be thrilled! It's nice if really active/enterprising customers can then author their own to expand it, but they shouldn't have to author their own to get a basic test done.

[uwedeportivo]

@dadlerj the problem is that they do have to provide which repos they want to add and which search queries they want to execute (etc). this varies from customer to customer. we could have them all use one of our sourcegraph-testing repos, but some of the customers don't even want or can't access github.com. they also need to provide their own github token even if they do have access to github.com.

that's why i thought this is a good compromise. it's simple, restricted enough but a good place to pass this information into the validation. and flexible enough that customers can use their own setup and tokens and decide which verifications and steps are important to them.

we could narrow it down even further with them providing a repo name, the github token etc in a config file. but explaining and documenting these pieces of information gets you almost to this scripting.

[beyang]

I support the idea of enabling customers to write their own validation scripts to exercise Sourcegraph after updating and think this is a great start. Here are my concerns:

• Is skylark the right language to commit to? I don't know anything about it beyond that it's Python-like and it's the language Google uses for Blaze/Bazel. It does not seem widely used outside of that.
• Should we first test this out more thoroughly ourselves before asking customers to invest time in writing their own validation scripts? Once customers start using it, we are on the hook for supporting it, which means maintaining it in the long-run or providing excellent assistance for migrating customers off of it if we abandon it in the future.

If the goal here is just to rapidly prototype this as a proof-of-concept and use it ourselves, then I would be okay approving this under the condition that it is very very clear that it is experimental and unsupported, and it could disappear or change in a breaking fashion at any time.

I would also like to see more examples of tests written using this subcommand that we use ourselves in our testing/release pipeline. That would help me determine if this is worth shipping to customers.

[uwedeportivo]

@beyang i agree with all your points. this is very much experimental. i needed a vehicle for deployment and chose src-cli. customers won't hear about it until we're sure we want to go down this road. if they discover it by accident then it's clearly marked as experimental. we won't advertise it.

i'm also not sure about the scripting language choice. i wanted something simple with a familiar syntax. we can plug in a different choice without too much effort.

i will use deploy-sourcegraph as the use case to kick the tires on it and test the approach.

[uwedeportivo]

sounds good. i'll call it out more and try to re-use what's there (there isn't much for adding and removing)

[daxmc99]

Looks good! You can see it running here https://buildkite.com/sourcegraph/deploy-sourcegraph/builds/3497#5af4349d-a259-40ea-9c6e-dd544b36caf0

[slimsag]

Remove and just have it print the URL as part of the help output instead? I don't understand why a -doc flag is useful when -h exists for this purpose :)

[slimsag]

Please add docs for this command to the handbook under distribution (or the actual docs if we want it public). Specifically the config format being documented would be nice.

[slimsag]

Awesome, looks really good to me!

[mrnugget]

What's the status here? I get pinged by the GitHub bot in Slack about this PR :)

[uwedeportivo]

@mrnugget it landed :-)

[mrnugget]

@uwedeportivo Can you add a CHANGELOG entry?