structure["key.offset"] is off by 1

[pkasson]

Kitten reports the offset for the function body to be off by 1, and chops off the beginning of the function name for the body:

unc numberOfComponents(in pickerView: UIPickerView) -> Int { return 1 }

Was parsing BitcoinTicker on Github and noticed all the function bodies are missing the first character.

[jpsim]

It would really help if you shared the steps you took to reproduce this issue. We haven't seen this before.

[pkasson]

Sorry, here is the snippet where it is retrieving data from the structure:

let name = structure["key.name"] let keykind = structure["key.kind"] let keyoffset = structure["key.offset"] let keylength = structure["key.length"] let accessibility = structure["key.accessibility"] let funcAttributes = structure["key.attributes"] let substructures = structure["key.substructure"]

A Swift class is being scanned, and the structure is being built (cast) as:

let structure = sourceKitRep as! [String : SourceKitRepresentable]

after the structure is read in:

structure = try Structure(file: File(path: filePath)!)

Once the structure is valued, then locating the function structures, the process gets the function bodies (this worked fine on OSX, but just noticed on Ubuntu this behavior):

var part = getPart(sourceFile: contents, start: keyoffset - 1, length: keylength + 1)

Using: SourceKitten 0.27.0, on Ubuntu 18 ...

Linux 4.15.0-1056 Ubuntu SMP Tue Nov 26 15:14:34 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

[Moriquendi]

I think I'm running into this issue.

Case 1 (no unicode chars)

// by Michal
func hello() {
    // Func body.
}

Output of sourcekitten structure --file test.swift:

{
  "key.diagnostic_stage" : "source.diagnostic.stage.swift.parse",
  "key.length" : 49,
  "key.offset" : 0,
  "key.substructure" : [
    {
      "key.accessibility" : "source.lang.swift.accessibility.internal",
      "key.bodylength" : 19,
      "key.bodyoffset" : 27,
      "key.kind" : "source.lang.swift.decl.function.free",
      "key.length" : 34,
      "key.name" : "hello()",
      "key.namelength" : 7,
      "key.nameoffset" : 18,
      "key.offset" : 13
    }
  ]
}

Case 2 (In the comment, the letter l is replaced with unicode ł).

// by Michał
func hello() {
    // Func body.
}

Output of sourcekitten structure --file test.swift:

{
  "key.diagnostic_stage" : "source.diagnostic.stage.swift.parse",
  "key.length" : 50,
  "key.offset" : 0,
  "key.substructure" : [
    {
      "key.accessibility" : "source.lang.swift.accessibility.internal",
      "key.bodylength" : 19,
      "key.bodyoffset" : 28,
      "key.kind" : "source.lang.swift.decl.function.free",
      "key.length" : 34,
      "key.name" : "hello()",
      "key.namelength" : 7,
      "key.nameoffset" : 19,
      "key.offset" : 14
    }
  ]
}

[jpsim]

Ah, the key.offset value is the offset in bytes into the file, not number of unicode code points or grapheme clusters or anything with semantic text value, just bytes.

[Moriquendi]

Ah, the key.offset value is the offset in bytes into the file, not number of unicode code points or grapheme clusters or anything with semantic text value, just bytes.

Hmm, that make sense.
@jpsim Any recommendations on how to obtain correct String.Index from the offset in bytes?

[jpsim]

You should use SourceKittenFramework, which has a number of methods for manipulating strings from byte offsets: https://github.com/jpsim/SourceKitten/blob/0.28.0/Source/SourceKittenFramework/StringView.swift#L133-L141

[Moriquendi]

@jpsim Thanks :)

[pkasson]

Would that be the cause of the issue ? Until I upgrade to Sahara a while back it worked fine as is ... but noticed since the upgrade that it is off at times.

[pkasson]

@Moriquendi - did using SourceKittenFramework fix things for you ?

[Moriquendi]

@pkasson @jpsim
I ended up with this code:

var fileString = try String(contentsOfFile: filePath, encoding: .utf8)

let stringView = StringView(fileString)
let mutableString = NSMutableString(string: stringView.nsString)

let swiftOffset = stringView.location(fromByteOffset: offset) // offset is a value from 'key.bodyoffset'
mutableString.insert(textToAppend, at: swiftOffset)


let modifiedFileData = String(mutableString).data(using: .utf8)!
// ... write to file

Looks like it's working so I hope it's correct 😬

[pkasson]

That helps !

Are you using Xcode strictly, or Swift Package Manager ?

I am having troubles upgrade to kitten 28, which includes StringView - totally hoses up the build.

[Moriquendi]

I used CocoaPods @pkasson

[pkasson]

Would you mind sharing the specific bits for your Podfile ?

Here is what I am using and its complaining about
source 'https://cdn.cocoapods.org/'
platform :osx, '10.14.5'
inhibit_all_warnings!

def shared_pods
pod 'SourceKitten', '~> 0.15.3'
pod 'SwiftyBeaver'
end

But the build complains about SWXMLHash being Swift 3 and that's coming from SourceKitten, how did you get it to work with Swift 4/5 ?

[Moriquendi]

@pkasson Ah, is your project a MacOS app or command line tool?

I think I had similar issue when I was trying to link those frameworks into command line tool - probably because you can’t have dynamic frameworks there. Not sure if there’s a way to force static linking in CocoaPods. If you do find the solution, please let me know 😬

[pkasson]

Yes, I usually try the unthinkable. I did get this thing to work fine using SPM and on Ubuntu no less, but upgrading one of the libraries made it sad.

However, by updating the paths to find all libraries, it ran just fine (not trying to distribute this thing, just run it).

[pkasson]

@Moriquendi

Switching from SPM to CocoaPods worked. BTW, if not deploying a CLI app, setting the paths allow the app to work, and do not need to be statically linked.

But, this is a good read on the topic:

https://medium.com/livefront/how-to-add-a-dynamic-swift-framework-to-a-command-line-tool-bab6426d6c31