[runtime] make swift_getFieldAt callable from C / Swift

[tanner0101]

Field names will no longer be stored in the nominal type descriptor in Swift 4.2. This PR allows the swift_getFieldAt function to be callable from C / Swift so that it will still be possible to access the field name data in-process from a Swfit app.

I've tested this from a Swift app and it works now. Prior to this change similar code would result in a bad access exception.

@_silgen_name("swift_getFieldAt")
func _getFieldAt(
    _ type: Any.Type,
    _ index: Int,
    _ callback: @convention(c) (UnsafePointer<CChar>, UnsafeRawPointer, UnsafeRawPointer) -> Void,
    _ ctx: UnsafeRawPointer
)

final class User {
    var name: String
    var age: Int
    init() { ... }
}

var test = "hello"

_getFieldAt(User.self, 1, { name, type, ctx in
    let name = String(cString: name)
    let type = unsafeBitCast(type, to: Any.Type.self)
    print("\(name): \(type)")
    print(ctx.assumingMemoryBound(to: String.self).pointee)
}, &test) // prints: age: Int, hello

Joe noted that I should add a context pointer to the callback which I did. He also said:

_swift_getFieldAt taking a std::function [should be] implemented in terms of the function-pointer-taking variant

I was a bit confused on how to implement it that way as I've never really used C++ before. I'm not quite sure how to pass the context-capturing std::function through a C closure. Any pointers there would be appreciated!

Related twitter thread: https://twitter.com/jckarter/status/978736605014892544

[slavapestov]

We really don't want people to use @_silgen_name and soon it might go away except for the stdlib. Can you use the C calling convention with @_cdecl instead?

[xedin]

I think Mirror is a preferred interface for reflection from Swift, you should probably look into using that instead of calling swift_getFieldAt directly, it's using that method internally?...

[slavapestov]

@xedin Mirror is implemented in C++ but I guess we can try to refactor it to expose an underscored method that can be called. However in any case this will be part of the ABI forever so we should at least pretend to design it properly.

[jrose-apple]

I'm with @xedin on this. Is swift_getFieldAt even supposed to be a part of the public ABI at the C/C++ level? cc @jckarter

(This is not to say you don't have legitimate uses for it, but if we want to expose it to Swift it should actually be designed and not just "whatever we needed for mirrors".)

[tanner0101]

I think Mirror is a preferred interface for reflection from Swift, you should probably look into using that instead of calling swift_getFieldAt directly, it's using that method internally?...

@xedin as far as I know there is no way to use Mirrors to get a type's fields. If there is a way, I would much rather use that instead.

We really don't want people to use @_silgen_name and soon it might go away except for the stdlib. Can you use the C calling convention with @_cdecl instead?

Hmm when I try @_cdecl it complains that it wants a function body. Do you have any examples on how to use that?

[jrose-apple]

*waves hands frantically* None of this "forward-declare a C function using only Swift" stuff is supported. @_silgen_name takes all the safeguards off in a way that's necessary for the stdlib, but not really something that works for users. If you did want to use something like this, the right way would be to import the declaration from C, just like you would for any other C function.

That said, Mirror.children is exactly this.

[jckarter]

There are things Mirror can't do that people need to do and have been doing unofficially by parsing metadata, particularly asking for the children of a type independent of a value of the type. As currently declared it is being exported from the runtime with a C++ calling convention, which is something we really don't want, so this is a strict improvement.

If you wanted to use this API from your own code outside of the standard library, though, you should declare swift_getFieldAt's prototype in a C header instead of using @_silgen_name.

[xedin]

@jckarter Why can't we just add this functionality to Mirror then if it's so popular? It seems reasonable that it should be able to expect not only values but types themselves, no?

[jckarter]

I think that would require fairly significant investment in Mirror, which is kind of an evolutionary dead end in its current form as far as reflection ability goes.

[jckarter]

Another thing that might make it easier for non-standard-library clients to interpret the metadata here would be to have nominal type context descriptors directly carry a reference to the associated field metadata structure, instead of registering them in parallel. @xedin IIRC you tried that; did it not work out for some reason?

[xedin]

Yes, I think it tried that but there was a problem related to circularity I think.

[paulofaria]

@jckarter @jrose-apple @slavapestov @xedin Would you say the best way forward then would be to create a swift evolution proposal to add proper reflection from the standard library?

[xedin]

@paulofaria I think @jckarter is trying to explore feasibility of adding this information into type context descriptors, if that doesn't work out we'd either have to accept your patch or figure out some other way to expose this information.

[TofPlay]

Any news about this pull request?
I am also interested to know if there have been any improvements in the Mirror class. Improvement that would allow me to avoid having to create a project like TProperties

[jckarter]

@swift-ci Please test

[swift-ci]

Build failed
Swift Test Linux Platform
Git Sha - 56713f7

[jckarter]

@swift-ci Please test linux

[swift-ci]

Build failed
Swift Test Linux Platform
Git Sha - 56713f7

[jckarter]

@swift-ci Please test Linux

[mtfourregp]

When do we expect to see this in an official Swift version?

[jckarter]

This was brought into the 4.2 branch in #17174 so it should be in 4.2 snapshots already, and eventually be in the official 4.2 release.

[robertjpayne]

For what it's worth in anyone ending up here this method of getting metadata is now unavailable as of the Swift 4.2 branch via #18746.

Still trying to sort out of there's a new workaround to resurface this information.

[jckarter]

#18746 was not committed to the 4.2 branch. You can still use this entry point in Swift 4.2. Anyone relying on Swift 4.2's metadata layout will have a lot more work in general moving to Swift 5's final layouts.

[lynnleelhl]

For Swift 5, do you have any suggestion if we still want to get the type's fields information as the previous version do?

[jckarter]

The field type metadata is directly referenced from the nominal type descriptor now. You can get the mangled type name from that data structure and feed it to the swift_getTypeByMangledNameInContext function with the struct or class's generic arguments in order to get the metadata pointer for the field type.

[lynnleelhl]

I solved it, thanks for the help :)

[paulofaria]

@jckarter I was able to get non-generic metadata by passing nil to genericContext and genericArguments. However, I would like to add support for generics. Can you give me any directions on how to get the context and arguments to properly feed the function?

[jckarter]

The generic context argument should be a pointer to the ContextDescriptor for the type whose fields you're decoding. The arguments should be a pointer to an array of metadata pointers followed by witness table pointers for any protocol requirements. For example, for Dictionary<T: Hashable, U>, you would pass the type context descriptor pointer for Dictionary along with an array of pointers containing:

• pointer to T's metadata
• pointer to U's metadata
• pointer to the protocol witness table for T: Hashable

If you already have metadata for a specific instance of the type, you should be able to extract the type descriptor pointer and generic arguments directly out of that metadata object. In the runtime C++ code, Metadata::getTypeContextDescriptor() will return the context descriptor pointer, and getGenericArgs will return a pointer to the generic arguments array.

[paulofaria]

Thank you for your fast response. Unfortunately, I still wasn't able to get it right. Is this still the layout for struct and class respectively?

struct StructMetadataLayout : MetadataLayoutType {
    var valueWitnessTable: UnsafePointer<ValueWitnessTable>
    var kind: Int
    var typeDescriptor: UnsafeMutablePointer<StructTypeDescriptor>
    var genericArgumentVector: UnsafeRawPointer
}

struct ClassMetadataLayout : MetadataLayoutType {
    var valueWitnessTable: UnsafePointer<ValueWitnessTable>
    var isaPointer: Int
    var superClass: Any.Type
    var objCRuntimeReserve1: Int
    var objCRuntimeReserve2: Int
    var rodataPointer: Int
    var classFlags: Int32
    var instanceAddressPoint: Int32
    var instanceSize: Int32
    var instanceAlignmentMask: Int16
    var runtimeReserveField: Int16
    var classObjectSize: Int32
    var classObjectAddressPoint: Int32
    var typeDescriptor: UnsafeMutablePointer<ClassTypeDescriptor>
    var genericArgumentVector: UnsafeRawPointer
}

I'm trying to pass the raw pointer of typeDescriptor and genericArgumentVector to swift_getTypeByMangledNameInContext. I also noticed that the type descriptors themselves also have a genericArgumentVector. What's the difference between the two? Thanks again.

[paulofaria]

Apparently, what I'm getting is a crash at swift_checkMetadataState when the property is generic. When the property is not generic the code runs fine.

[jckarter]

The generic argument vector is stored inline as a trailing field of the metadata record, not as a pointer. It looks like you have its relative position within the struct correct, but the address of that field would be the address of the generic arguments you want to pass.

The best way to get the offset would be to follow the logic of TargetTypeContextDescriptor::getGenericArgumentOffset in the runtime; particularly for classes, the exact offset can vary with vtable layout and the position of a class in the subclass hierarchy, since a subclass's metadata is also an instance of the superclass's metatype.