Support Xcode 6 GM

[jpsim]

Unfortunately, the approach we took with Xcode 6 Betas 1 & 2 no longer work with Beta 3.

This still works:

$ xcrun sourcekitd-test -req doc-info SwiftFile.swift

but this no longer works:

$ xcrun sourcekitd-test -req doc-info -module MyModule -- -I `pwd` -sdk `xcrun --show-sdk-path`
sourcekit: [1:connection-event-handler:271: 0.0000] Connection interrupt
sourcekit: [1:pingService:271: 0.0016] pinging service
error response (Connection Interrupted): Connection interrupted

Seems to be an issue with sourcekitd-test's XPC connection with SourceKitService, but I'm unsure how to fix it. If anyone has any ideas, I'd ❤️ to hear from you!

[neonichu]

Will have a look, wanted to play a bit more with SourceKit anyway.

[shepting]

Any news on Beta3 support?

[jpsim]

This issue is still open and I haven't found a workaround for Xcode6-Beta3.

If anyone has any experience writing XPC clients, we could connect to SourceKitService directly without relying on sourcekitd-test.

I looked into it briefly and it appears that @protocol's have to be created that match exactly what the XPC service is using, so it looks tricky to reverse engineer.

[zwaldowski]

Just briefly looking at it, it looks like SourceKitService.xpc isn't using Foundation XPC API, and is bridging straight from XPC/GCD/C into LLVM and C++ stuff. The XPC C Services API isn't compatible with the Foundation one, right? So would that mean Xcode is using the C API too?

[jpsim]

Thanks for looking at it, @zwaldowski. Whichever API it's using, we should be able to connect to it.

Here's an SourceKit XPC logging dump of Xcode6-Beta3: https://gist.github.com/jpsim/e5398e40047492d85966

Hopefully looking at that will help us connect to SourceKitService.

[zwaldowski]

If you create a target and link against sourcekitd.framework (/Applications/Xcode6-Beta3.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/sourcekitd.framework), you'll be able to connect to SourceKitService like so (forgive me, I'm not familiar with the XPC C Services API):

    dispatch_queue_t q = dispatch_queue_create("com.dizzytechnology.XPCTest.gcd", 0);
    xpc_connection_t conn = xpc_connection_create("com.apple.SourceKitService", q);
    xpc_connection_set_event_handler(conn, ^(xpc_object_t object) { });
    xpc_connection_resume(conn);

    xpc_object_t dict = xpc_dictionary_create(0, 0, 0);
    xpc_dictionary_set_bool(dict, "ping", true);
    xpc_connection_send_message_with_reply(conn, dict, dispatch_get_global_queue(0, 0), ^(xpc_object_t object) {
        if (xpc_get_type(object) == XPC_TYPE_ERROR) {
            const char *string = xpc_dictionary_get_string(object, XPC_ERROR_KEY_DESCRIPTION);
            NSLog(@"error: %s", string);
        } else {
            NSLog(@"Hello from SourceKit! They were nice enough to send us an empty dictionary! %@ %@", conn, dict);
        }
    });

A quick look at disassembly pseudo-code indicates:

• sourcekitd.framework uses the XPC C Services API, so we would have to as well, since they are incompatible with NSXPCConnection. (A higher-level equivalent could be achieved using something like the good old XPCKit.)
• SourceKitService accepts two messages: @{ "ping": @YES } (as above), returning @{} an empty dictionary, and @[ @"msg": @[ … ] }, returning @{ "response": @{ … }} and would correspond to some of what you see in that log file.

[jpsim]

This is really great, @zwaldowski! I've managed to reproduce your setup and the ping works. Now it shouldn't be too hard to call "source.request.docinfo" from here.

[philipengberg]

Any status on this issue?

[jpsim]

@Zappel I haven't worked on this since the last comment 2 weeks ago. The biggest challenge now is to create an XPC client that connects to SourceKitService to call source.request.docinfo. It's a matter of reverse-engineering the XPC interface to craft the exact format the call needs to have in order to succeed.

[guillaumealgis]

I dived a bit in the assembly of sourcekitd today, and building on @zwaldowski 's comment, I managed to have a proper response from SourceKit.

#import <Foundation/Foundation.h>

void sourcekitd_initialize();
uint64_t sourcekitd_uid_get_from_cstr(const char *);
xpc_object_t sourcekitd_send_request_sync(xpc_object_t);

int main(int argc, const char * argv[])
{
    sourcekitd_initialize();

    xpc_object_t request = xpc_dictionary_create(NULL, NULL, 0);
    xpc_dictionary_set_uint64(request, "key.request", sourcekitd_uid_get_from_cstr("source.request.docinfo"));

    // Use this parameter to parse from file
    xpc_dictionary_set_string(request, "key.sourcefile", "/Users/guillaume/Projets/jazzy/sample/Musician.swift");

    // Or this one to parse from string
//    xpc_dictionary_set_string(request, "key.sourcetext", "//\n//  Musician.swift\n//  JazzyApp\n//\n\n/**\n Musician models jazz musicians.\n From Ellington to Marsalis, this class has you covered.\n*/\nclass Musician {\n    /**\n     The name of the musician. i.e. \"John Coltrane\"\n    */\n    var name: String\n\n    /**\n     The year the musician was born. i.e. 1926\n    */\n    var birthyear: UInt\n\n    /**\n     Initialize a Musician.\n     Don't forget to have a name and a birthyear.\n\n     @warning Jazz can be addicting.\n     Please be careful out there.\n\n     @param name      The name of the musician.\n     @param birthyear The year the musician was born.\n\n     @return          An initialized Musician instance.\n    */\n    init(name: String, birthyear: UInt) {\n        self.name = name\n        self.birthyear = birthyear\n    }\n}");

    xpc_object_t result = sourcekitd_send_request_sync(request);
    if (xpc_get_type(result) == XPC_TYPE_ERROR) {
        const char *string = xpc_dictionary_get_string(result, XPC_ERROR_KEY_DESCRIPTION);
        NSLog(@"error: %s", string);
    } else {

        NSLog(@"%@", result);
    }
}

I'm using 3 functions directly from the sourcekit framework to work around guessing the format of the messages. Another win when using these is having the nice logging output when SOURCEKIT_LOGGING is defined in the environment.

I'm a bit frustrated as I could not figure out what to put in the array of @{ @"msg": @[ … ] }, but at least it's working :).

[shepting]

Perhaps we should update the title to Beta 5?

[jlukas]

You can also do it via FFI:

require 'pathname'

require 'ffi'

module LibC
  extend FFI::Library
  ffi_lib FFI::Library::LIBC

  attach_function :malloc, [:size_t], :pointer
  attach_function :free, [:pointer], :void
end

module XPC
  extend FFI::Library
  ffi_lib '/usr/lib/system/libxpc.dylib'

  # [array of string, array of xpc_object_t, size_t], xpc_object_t
  attach_function :dictionary_create, :xpc_dictionary_create, [:pointer, :pointer, :size_t], :pointer
  # [xpc_object_t, string, uint64_t], void
  attach_function :dictionary_set_uint64, :xpc_dictionary_set_uint64, [:pointer, :string, :uint64], :void
  # [xpc_object_t, string, string], void
  attach_function :dictionary_set_string, :xpc_dictionary_set_string, [:pointer, :string, :string], :void
  # [xpc_object_t, string], string
  attach_function :dictionary_get_string, :xpc_dictionary_get_string, [:pointer, :string], :string
  # [xpc_object_t], xpc_type_t
  attach_function :get_type, :xpc_get_type, [:pointer], :pointer
  # [xpc_object_t], void
  attach_function :release, :xpc_release, [:pointer], :void
  # [xpc_object_t], xpc_object_t
  attach_function :retain, :xpc_retain, [:pointer], :pointer
  # [xpc_object_t], string
  attach_function :copy_description, :xpc_copy_description, [:pointer], :string

  module Type
    extend FFI::Library
    ffi_lib '/usr/lib/system/libxpc.dylib'

    attach_variable :error, :_xpc_type_error, :pointer
  end

  module ErrorKey
    extend FFI::Library
    ffi_lib '/usr/lib/system/libxpc.dylib'

    attach_variable :description, :_xpc_error_key_description, :string
  end
end

module SourceKitD
  extend FFI::Library

  developer_dir = Pathname(`xcode-select -p`.chomp)
  dt_toolchain_dir = developer_dir + 'Toolchains' + 'XcodeDefault.xctoolchain'
  ffi_lib dt_toolchain_dir + 'usr' + 'lib' + 'sourcekitd.framework' + 'sourcekitd'

  attach_function :sourcekitd_initialize, [], :void
  sourcekitd_initialize

  attach_function :uid_get_from_cstr, :sourcekitd_uid_get_from_cstr, [:string], :uint64
  # [xpc_object_t], xpc_object_t
  attach_function :send_request_sync, :sourcekitd_send_request_sync, [:pointer], :pointer
end

source = <<SOURCE
//
//  Musician.swift
//  JazzyApp
//

/**
 Musician models jazz musicians.
 From Ellington to Marsalis, this class has you covered.
*/
class Musician {
    /**
     The name of the musician. i.e. \"John Coltrane\"
    */
    var name: String

    /**
     The year the musician was born. i.e. 1926
    */
    var birthyear: UInt

    /**
     Initialize a Musician.
     Don't forget to have a name and a birthyear.

     @warning Jazz can be addicting.
     Please be careful out there.

     @param name      The name of the musician.
     @param birthyear The year the musician was born.

     @return          An initialized Musician instance.
    */
    init(name: String, birthyear: UInt) {
        self.name = name
        self.birthyear = birthyear
    }
}
SOURCE
request = XPC.dictionary_create(nil, nil, 0)
XPC.dictionary_set_uint64(request, "key.request", SourceKitD.uid_get_from_cstr("source.request.docinfo"))
XPC.dictionary_set_string(request, "key.sourcetext", source)

result = SourceKitD.send_request_sync(request)
if XPC.get_type(result) == XPC::Type.error
  error = XPC.dictionary_get_string(result, XPC::ErrorType.description)
  puts "error: #{error}"
else
  desc = XPC.copy_description(result)
  puts desc
  LibC.free(desc)
end
XPC.release(request)
XPC.release(result) # not entirely sure about this one

[jpsim]

I'm making small incremental progress on this issue, thanks to the efforts of everyone on this thread.

In a proof-of-concept project, we can now extract documentation information for Swift source files, SDK modules, Swift frameworks... but not Objective-C frameworks. This last one is the last holdout. If you'd like to give it a shot you can fork the SourceKitten sample project and try to make approach #3 work.

[zwaldowski]

@jpsim Looking at the debugging output for approach #2 (parsing Foundation) and trying to play around with it, sourcekitd seems to only dump the syntax info for the Swift view of Foundation. Same for parsing the Swift framework; it's only able to get the source info because it synthesizes it from the *.swiftmodule and not the module map. These both lead me to believe that sourcekitd still doesn't support Objective-C at this point in the game.

[jpsim]

You're quite right, although running docinfo on a modulemap used to work with sourcekitd-test, leading me to believe that this behavior still exists.

Also, command-clicking on an objective-c USR from Swift in Xcode does trigger relevant sourcekit logs.

It might be necessary for us to build a swiftmodule from Objective-C in order to run docinfo on it. That's the next thing I'm trying.

[jpsim]

Good news: we've regained the ability to generate a Swift interface from an Objective-C header! See the SourceKitten project for details.

Bad news: we don't have USR's for the swift tokens in the generated Swift interface. But there's still hope. I think if we pass in the USR in the request, the equivalent Swift token will be "highlighted" in SourceKit's response.

But nonetheless, this was the major hurdle to getting jazzy back on track, so I hope to be able to bring it back up and running in the next few days.

[Aymenworks]

Hi,
Any update ?

[beltex]

I guess at this point it's probably best just to wait for GM next week? As it doesn't seem like we'll be getting a beta 7.

[jpsim]

Work that we do to support beta 6 will likely be transferable to GM, at the very least partially, so we'll keep working with the latest beta until then.

[beltex]

Good point.

And I spoke too soon, we have a beta 7 now. :)

[Aymenworks]

Thanks you all. Impatient to test it !

[Reefaq]

Eagerly waiting for the fixes. any estimate when this will be made available?

[Aymenworks]

I want it too.. The apple template style documentation is so good ..

[jpsim]

Hi everyone, a status update is long overdue. Here's where we are:

• We've had to completely rethink how jazzy works due to Apple changing what tools ship with Xcode 6.
• We're rebuilding jazzy to use a direct XPC connection to SourceKit, which although is likely to change, is unlikely to disappear in the future. So this is a more future-proof approach.
• Progress on SourceKit's XPC interface can be loosely monitored at jpsim/SourceKitten. Once this tool meets jazzy's needs, we'll clean it up and release it on its own, so others can build tools that connect to SourceKit too.

Roadmap

jazzy's goal is to be a cross-language documentation tool, which allows generating mixed Objective-C/Swift documentation from projects written in Objective-C, Swift or a mix of both. Here's a tentative roadmap including planned features for each release:

Release	Project Support	Documentation Output
0.1.0	Swift Only	Swift Only
0.2.0	Objective-C Only	Objective-C Only
0.3.0	Objective-C & Swift	Objective-C & Swift

0.1.0

• generate docs for self-contained Swift files (files can't reference other files, imports not allowed)
• generate docs for Swift files only referencing other modules (files can't reference other files, imports allowed)
• generate docs for Swift modules (anything Xcode 6 can build: multiple files, imports allowed)

0.2.0

• generate docs for any Objective-C header

0.3.0

• find Objective-C token from Swift USR
• find Swift token from Objective-C USR

Final thoughts

Because jazzy uses internal Apple tools, there's an up-front investment in reverse engineering that we have to make. But once that's done, the payoff will be huge since we won't have to maintain a complex parser.

Around 0.1.0, I'll make an effort to clean up and document the codebase (oh, the irony) so others can more easily contribute. Until then, thanks for your patience!

[guillaumealgis]

👍 I'll take a look at SourceKitten (great name btw) this week, I started to dig into the sourcekitd_* functions a while ago.

[jpsim]

The march of progress continues! Documentation comments can now be extracted from any project that Xcode 6 can build. See this branch of SourceKitten and this sample output gist.

Now it's a question of integrating this back into jazzy. Thanks for your patience.

[EmperiorEric]

I don't have much to provide in the form of help, but I love what you guys are doing and the work going into this project.

[mrh-is]

I see an exciting flurry of activity! Does that mean that Jazzy for Xcode 6.0/6.1 is coming soon?

[jpsim]

Hi everyone, thanks so much for your patience. jazzy 0.0.5 was just released!

As a demo, I've generated Alamofire 1.1.0's docs, visible here: http://static.realm.io/jazzy_demo/Alamofire/

If you've been hoping to contribute to jazzy, now's a great time to start. Try it with your project and see what works, what doesn't. Please file issues when you find scenarios that don't work.

Areas of focus that I'd encourage others to provide feedback and fixes are in the front-end HTML and updating the misc_jazzy_features integration test with more scenarios, both supported and unsupported.

Big thanks to @segiddins for setting up a good ruby foundation with integration tests, rubocop and better encapsulation. Also a big thanks to everyone in this issue for your help, support and patience as we worked on getting jazzy back on track.