[SR-2487] sourcekitd doesn't include __raw_doc_comment in structure

[keith]

Previous ID	SR-2487
Radar	None
Original Reporter	@keith
Type	Bug
Status	Resolved
Resolution	Done

Additional Detail from JIRA

Votes	14
Component/s	Source Tooling
Labels	Bug, Documentation
Assignee	@marcelofabri
Priority	Medium

md5: 7cb28eacb8646ac198576beb5bef2cf8

Issue Description:

Previously (with Xcode 7.x) if you had a property with a docstring (like this):

struct Foo {
    /// Some var
    var bar: String
}

And you got its structure for sourcekit (using sourcekitten) the output would look like this:

% sourcekitten structure --file foo.swift
{
  "key.substructure" : [
    {
      "key.kind" : "source.lang.swift.decl.struct",
      "key.offset" : 0,
      "key.nameoffset" : 7,
      "key.namelength" : 3,
      "key.bodyoffset" : 12,
      "key.bodylength" : 38,
      "key.accessibility" : "source.lang.swift.accessibility.internal",
      "key.substructure" : [
        {
          "key.kind" : "source.lang.swift.decl.var.instance",
          "key.offset" : 34,
          "key.attributes" : [
            {
              "key.attribute" : "source.decl.attribute.__raw_doc_comment"
            }
          ],
          "key.nameoffset" : 38,
          "key.namelength" : 3,
          "key.length" : 15,
          "key.accessibility" : "source.lang.swift.accessibility.internal",
          "key.setter_accessibility" : "source.lang.swift.accessibility.internal",
          "key.typename" : "String",
          "key.name" : "bar"
        }
      ],
      "key.name" : "Foo",
      "key.length" : 51
    }
  ],
  "key.offset" : 0,
  "key.diagnostic_stage" : "source.diagnostic.stage.swift.parse",
  "key.length" : 52
}

Notice this attribute:

            {
              "key.attribute" : "source.decl.attribute.__raw_doc_comment"
            }

With the Xcode 8.x betas, or building from the swift repo, the output for the same source now looks like this:

% ./sourcekitd-test -req=structure foo.swift
{
  key.request: source.request.editor.open,
  key.name: "foo.swift",
  key.sourcefile: "foo.swift",
  key.enablesyntaxmap: 0,
  key.enablesubstructure: 1,
  key.syntactic_only: 1
}
{
  key.offset: 0,
  key.length: 52,
  key.diagnostic_stage: source.diagnostic.stage.swift.parse,
  key.substructure: [
    {
      key.kind: source.lang.swift.decl.struct,
      key.accessibility: source.lang.swift.accessibility.internal,
      key.name: "Foo",
      key.offset: 0,
      key.length: 51,
      key.nameoffset: 7,
      key.namelength: 3,
      key.bodyoffset: 12,
      key.bodylength: 38,
      key.substructure: [
        {
          key.kind: source.lang.swift.decl.var.instance,
          key.accessibility: source.lang.swift.accessibility.internal,
          key.setter_accessibility: source.lang.swift.accessibility.internal,
          key.name: "bar",
          key.offset: 34,
          key.length: 15,
          key.typename: "String",
          key.nameoffset: 38,
          key.namelength: 3
        }
      ]
    }
  ]
}

Which doesn't include the `__raw_doc_comment` attribute.

I also noticed that if I use the sourcekitd-repl and pass `key.enablesyntaxmap: 1`, it does provide some information about the doccomment:

    {
      key.kind: source.lang.swift.syntaxtype.doccomment,
      key.offset: 17,
      key.length: 13
    },

But it is in a top level array of elements, and not directly associated with the key that it's documenting.

Is this new behavior expected? Or is this a bug?

[keith]

It looks like this was intentionally removed, but I don't see much context on why. c78f169 @akyrtzi is there any way we could make this more production ready so we can rely on it again in linters and other tools?

[norio-nomura]

SourceKitten has been affected by this change.
jpsim/SourceKitten#269

[swift-ci]

Comment by David Potter (JIRA)

This hurts our ability to enforce documentation on public definitions, including those frameworks that are targeted for external SDKs.

[jpsim]

Some visibility into the motivations for this change would be greatly appreciated.

[swift-ci]

Comment by Mohamed Afifi (JIRA)

I would love to get this back since it would be very helpful in linting tools like Swiftlint.

[swift-ci]

Comment by Hristo Todorov (JIRA)

It would be crucial for us to understand if something is going to replace the old attribute so we can still use lint tools.

[swift-ci]

Comment by Stefan Herold (JIRA)

I have trouble with SwiftLint too. It reports a lot of false positives because it cannot recognise already documented code anymore since this issue has arisen.

[swift-ci]

Comment by Yen-Chia Lin (JIRA)

I'm having issues with SwiftLint too. Produces lots of false positives, so I had to turn off that SwiftLint rule. I'm using Xcode 8.3.2

[keith]

I've submitted a simple revert PR of the commit that removed this #9868

[marcelofabri]

I've submitted a PR that adds `key.docoffset` and `key.doclength`: #11264

[keith]

Thanks for doing this @marcelofabri!

Do we know if this will be part of the Swift 4 release, or if it will come later?

[marcelofabri]

This was merged to master, so I don't think it'll be included in Swift 4.