Output spec type, name, and codelocation - for use by an IDE

[dlipovetsky]

Once there are more than a few specs in a file, navigating to a particular spec, or getting an overview of all the specs, is not easy manually, and is not possible (as far as I know) with an IDE, e.g., VSCode.

Take, for example, https://github.com/onsi/ginkgo/blob/efb9e6987c00c280f4c3c624a105a9f6c27f2122/internal/spec/spec_test.go. If each spec were a standard go test, it would included in the list of top-level symbols, because the test is a function, e.g. TestThisFeature. In Ginkgo, a spec is a function call, e.g., It(...).

It's worth noting that both standard go tests and ginkgo specs can be defined at runtime, e.g., as is the case with table-driven tests. I think it would still be helpful to quickly navigate to the line where that happens.

Here's a comparison of navigating to a symbol in VSCode, using a test from the standard librarytime package, and a test from the internal/spec package:

I'd like to be able to output a list or tree structure with all of the containers and specs, their types, text, and code locations, for a specific file. That would be enough information to provide to an IDE like VSCode, where an extension could make use of it.

[dlipovetsky]

I've tried two experiments to generate the tree of all containers and specs, their types, text, and code locations, for a specific file.

1. Parse the file and walk the AST. Upsides: No dependency on ginkgo. Downsides: Might not work with your ginkgo version, if the syntax changes (unlikely). It's not straightforward to get the text of a spec; although it's almost always a string literal, but sometimes it is not.
2. Re-use the tree built by internal/suite. Upsides: Could be a new ginkgo command ,e.g., ginkgo outline. Guaranteed to work with your ginkgo version. Easy to get the tree! Downsides: test code must be compiled and evaluated to generate the tree. The internal tree structure doesn't make the spec types available.

[onsi]

@dlipovetsky I love this idea and I'm super supportive of making it a reality - though I don't have a ton of time on my hands to make it so these days.

I agree that a new ginkgo sub-command would make sense here. And I think it's reasonable for a plugin for Ginkgo tests to take a hard dependency on the ginkgo cli and either the tree-parsing approach or the AST approach could be folded into that subcommand. I'd be inclined to start with the AST approach as that would have a chance of working when the suite is malformed and cannot formally compile (e.g. you're working on a test in one file and leave it in a half-broken state but then want to navigate to ta different test in a different file within the same package). As you say, the text of a spec is not always a string literal but I think that edge case is OK and would be intuitive to most users/developers.

So... a sub-command that takes a package or lists of packages as an argument and spits out a JSON description of the tree would be a great addition. Are you interested in working on a PR? I'd be happy to merge it in and try to help - and we could drive it out with a few integration tests to ensure the functionality doesn't break going forward.

[dlipovetsky]

Thanks for the feedback, @onsi!

I'd be inclined to start with the AST approach as that would have a chance of working when the suite is malformed and cannot formally compile ...

+1

As you say, the text of a spec is not always a string literal but I think that edge case is OK and would be intuitive to most users/developers.

+1

Are you interested in working on a PR?

Yes. I've got a first iteration AST approach working. I'll clean it up a little, wrap a sub-command around it, PR it, and continue to iterate on that PR.

Sample input:

package p

import (
        "fmt"

        . "github.com/onsi/ginkgo"
)

var _ = Describe("group 1", func() {
        Context("context 1", func() {
                It("should test 1.1", func() {
                })
                It("should test 1.2", func() {
                })
        })
        Context("context 2", func() {
                It("should test 2.1", func() {
                })
                It(fmt.Sprintf("should test %d.%d", 2, 2), func() {
                })
        })
})

Sample output. (I'm constructing a tree, because that can be easily mapped to a "tree view," and that kind of view makes it easy to see the grouping of specs. This is just the tree marshalled to JSON, with indentation. Other outputs are possible.)

{
  "SpecType": "root",
  "CodeLocation": "",
  "Text": "",
  "Children": [
    {
      "SpecType": "Describe",
      "CodeLocation": "src.go:9:9",
      "Text": "group 1",
      "Children": [
        {
          "SpecType": "Context",
          "CodeLocation": "src.go:10:2",
          "Text": "context 1",
          "Children": [
            {
              "SpecType": "It",
              "CodeLocation": "src.go:11:3",
              "Text": "should test 1.1",
              "Children": null
            },
            {
              "SpecType": "It",
              "CodeLocation": "src.go:13:3",
              "Text": "should test 1.2",
              "Children": null
            }
          ]
        },
        {
          "SpecType": "Context",
          "CodeLocation": "src.go:16:2",
          "Text": "context 2",
          "Children": [
            {
              "SpecType": "It",
              "CodeLocation": "src.go:17:3",
              "Text": "should test 2.1",
              "Children": null
            },
            {
              "SpecType": "It",
              "CodeLocation": "src.go:19:3",
              "Text": "[could not determine]",
              "Children": null
            }
          ]
        }
      ]
    }
  ]
}

[onsi]

That looks great @dlipovetsky - thanks so much for working on this :)

A couple of quick thoughts:

• I think each node in the tree could also describe whether it is focused or pending. It is often useful to "find all pending specs" or "take me to the focused test."
• There are a few aliases in Ginkgo (e.g. Specify for It and When for Context and Describe) - those'll need to get captured as well.
• I think rather than "[could not determine]" simply setting Text to null would convey the same notion, no?
• If possible, I think it would also be useful to capture any By statements as children of It nodes.

I'm working on a Ginkgo 2.0 and I don't think anything in the works will break this output format. One thing we'll be adding is support for tags - and I imagine it'll be nice for tags to also appear in the tree view json structure... but, again, that's a future thing :)

[dlipovetsky]

Thanks for this feedback. All good ideas!

• I think each node in the tree could also describe whether it is focused or pending. It is often useful to "find all pending specs" or "take me to the focused test."

Done, including backpropagation of unfocus, and propagation of inherited focus/pending property 😅

• There are a few aliases in Ginkgo (e.g. Specify for It and When for Context and Describe) - those'll need to get captured as well.

Done.

• I think rather than "[could not determine]" simply setting Text to null would convey the same notion, no?

I decided to use undefined for now, but maybe null connotes the right thing, i.e., "don't know."

• If possible, I think it would also be useful to capture any By statements as children of It nodes.

Done.

[dlipovetsky]

Documentation PR: #757

[dlipovetsky]

Add support for the "table" extension: #758

[dlipovetsky]

I'm going to call this done! Thanks for the feedback and reviews, @onsi!

[onsi]

hey @dlipovetsky this outline stuff is great - and i'm excited to play with your vscode plugin when you're ready to share ;)

i'm wondering if you'd be interested in joining as a Ginkgo & Gomega committer? Can always use help engaging with issues and PRs and investing in Ginkgo and Gomega - any thoughts?

[dlipovetsky]

@onsi My apologies--I missed your last message! I just came back to this issue to share an update.

Development took a bit, but I made steady progress over the past month, and I just published the extension! It's got an outline view and a go-to-symbol menu. I'd love to get your feedback. Two ways to install:

1. Marketplace: https://marketplace.visualstudio.com/items?itemName=dlipovetsky.ginkgo-tools
2. Extension .vsix file: https://github.com/dlipovetsky/vscode-ginkgo-tools/releases/ (in the release artifacts)

Honestly, I'm flattered that you'd ask if I'd be interested to become a committer! What kind of expectations do you have for committters? I just want to make sure I could meet them.

[onsi]

hey @dlipovetsky - super cool! I'll take a look 😄

there aren't any formal expectations or anything - I'd say:

• help triage and respond to issues and PRs as they come in
• help maintain code, especially code you may have contributed to
• periodically collaborate on cutting a new release

in addition there's a ginkgo channel on the gophers slack instance as well as a ginkgo-internal channel that some of us use to chat about ginkgo. both are low traffic and you'd be welcome to join either but wouldn't have to.

right now i'm heads down making headway on v2 - that's going to pull in a lot of changes in the coming weeks/months as i start to put it in a public branch and, eventually, merge things into master so there may be a fair bit of activity in the near-term but i expect things will level off after that.

thoughts?