[SR-363] SourceKit interface generation test chokes on non-ASCII characters

[belkadan]

Previous ID	SR-363
Radar	None
Original Reporter	@belkadan
Type	Bug

Additional Detail from JIRA

Votes	0
Component/s	Source Tooling
Labels	Bug
Assignee	@akyrtzi
Priority	Medium

md5: 74e1fbbee14341479e41fb4410fa4713

Issue Description:

If doc comments contain Unicode characters, sourcekitd-test just gives up on generating an interface.

{
  key.request: source.request.editor.open.interface,
  key.name: "/<interface-gen>",
  key.modulename: "Swift"
}
!!Interface text is non-ascii!!
@ 1013:30

This means our standard library comments are suboptimal.

[belkadan]

Once this is no longer an issue we can revert 27e38fa.

[lilyball]

Skimming through the source, it appears that this ASCII check is actually controlled by the flag -check-interface-ascii to sourcekitd-test. And our test test/SourceKit/InterfaceGen/gen_stdlib.swift provides this flag. So the question is, why do we have this check at all?

[belkadan]

I assume it's to make sure we don't accidentally mangle something, or maybe Xcode's side of the channel isn't set up to handle it. @akyrtzi?

[akyrtzi]

There is performance degradation if the stdlib interface contains non-ASCII, that's why the test exists.
We have a bug report internally to improve on this, and we could keep this as the JIRA side.

In the meantime, is there a particular reason that ASCII cannot be used instead of unicode in the stdlib comments ?

[belkadan]

Well, it doesn't always mean the same thing. In the commit referenced above, I changed … (an ellipsis character) to ... (three periods) within an inline code span, but ... is an actual thing you type in Swift, so it could be confused for that.

Personally, it seems ridiculous to me to limit ourselves to ASCII.

[lilyball]

What @belkadan said. I used an ellipsis precisely because it made it much more obvious that the ellipsis was a stand-in for omitted code rather than being an actual instance of the ... operator.

Also, since Swift supports unicode operators, eventually we might want to actually define one of those, and then we'd have no choice but to have unicode in the stdlib interface.

[akyrtzi]

In case anyone is interested in helping improve on this, the primary work is enhancing the SourceKit responses, for syntax/semantic annotations and document structure, to return line/column pairs for source locations instead of byte offsets.
For backwards compatibility transitioning, both byte offsets and line/column pairs should be provided.

[lilyball]

Doing column pairs instead of source offsets means knowing the actual width of each character. Is it reasonable to actually know that? Because the rendered width depends on the system doing the rendering (and how up-to-date that system's knowledge of Unicode is). I know there's a function wcswidth for determining the width of characters, but that depends on how up-to-date the Unicode support is in the system's libc. There's also the question of how to handle East Asian Ambiguous characters (According to UAX #11, when used in the context of the legacy encoding that they come from, they should be treated as fullwidth or halfwidth based on that context, otherwise they should be treated as narrow, although apparently Ambiguous quotation marks are a little more complicated).

Although I guess at this point I should ask how swiftc handles printing error indicators for lines that contain non-ASCII (either grapheme clusters, or fullwidth characters, or Ambiguous characters). Does it get it right always or do we have examples of the error indicator being off?

It's also worth nothing that, at least a while ago, Rust had a problem with error indicators in lines that contain non-ASCII (because of this issue). I don't know if they ever solved it completely. I'm not sure if it even can be solved completely, if for no other reason than the fact that Terminal.app has an explicit setting for whether Ambiguous characters should be wide or narrow (though it defaults to narrow, and it's probably a reasonable assumption that it should be narrow).

Taken all together, I wonder if maybe it would be more useful for SourceKit to report a unicode scalar offset (it can still report line too, but it could use unicode scalar offset within that line instead of column offset). That way the front-end can decide how best to handle this, e.g. Xcode knows exactly how it renders text and can therefore have precise indicators, whereas a CLI tool like swiftc, when emitting human-readable output, can make assumptions such as wcswidth() being appropriate and Ambiguous characters being narrow (which is to say, make assumptions that would produce accurate output on Terminal.app).

[akyrtzi]

Sorry it wasn't clear, I meant pairs of line/byte-offset-in-line, not line/column-of-character. The byte-offset will be for UTF8 which is what the swift compiler uses.

[Dante-Broggi]

Has this been fixed? If so, this should be closed.