Easier embedding of Rust / into Rust

[llogiq]

Hi there. I'm experimenting with embedding koto in another crate, and I feel the protocol around ExternalVaues (as showcased in the example; to my knowledge there are no other docs) is quite cumbersome.

I have built a procedural macro that can insert Rust functions into a given prelude, however, I'd like to also extend it to derive support for my own types (perhaps in conjunction with the proc macro also setting up a kind of vtable). What do you think about simplifying this interface to an ExternalValue + a Box<dyn fn(KeyValue) -> ExternalFunction> that will do the vtable dispatch? This might both be faster and easier to code against.

[irh]

Hi @llogiq, glad to hear that you're trying out Koto!

I'm sorry that the docs are missing here, I started a documentation effort on the help branch but I haven't touched the topic of integrating with Rust yet and I've had a busy few weeks, I'm looking forward to picking this up again.

I agree that making custom types is a bit cumbersome at the moment, especially without a guide.. My thinking is that I'd like to make sure the fundamentals are correct and the implementation straight-forward, and then some more ergonomic helpers can be added to make usage a bit friendlier.

I'll try to clarify the current approach:

• An ExternalValue is just a placeholder for some arbitrary Rust value. The intention is that scripts don't interact with them directly, but instead to make them useful they get placed in a ValueMap as a hidden value using ExternalDataId.
• An interface is then built up around the external value in the ValueMap. Functions and any other value type can be added to the map, like with any other map in Koto.
• There are a couple of other examples in the core library that can be looked at. The File example is more extensive. You can see that I added a file_fn helper for reducing boilerplate when adding functions.

If I understand you correctly you'd rather expect that the ExternalValue itself would be the value that the user interacts with (instead of a ValueMap), with behaviour on the value exposed via a KeyValue -> Function mapper?

If I've got that right I think I would rather extend the type via an embedded ValueMap (just to enable all value types, not just functions). The difference in approaches would then be that you would have an ExternalValue with an embedded ValueMap, rather than a ValueMap with an embedded ExternalValue...

Does that make sense? I can see some value in inverting the approach here, it might make the expected usage more explicit (although maybe utilities and docs will help there).

[llogiq]

Adding a ValueMap field in the ExternalValue variant makes a lot of sense; those are Arc and can easily be generated as a lazy static to be cloned for further use.

[irh]

Yes I can see an advantage there, it'd be cheaper to clone the interface map than having to rebuild it for each new value.

[llogiq]

I have started a WIP PR #64. Do you think this is the right direction?

[irh]

Yes I think this is roughly how I would expect it to look, I appreciate you diving in!

I'm still unsure if I want to proceed in this direction though. The main concern I have is that this will introduce a new object-like-thing to Koto, and I’d like to avoid that if possible.

I'm thinking about an alternative solution which would be to extend ValueMap so that it includes an optional ExternalValue, plus an optional ‘immutable data’ map, which would act as fallback when values aren't found in the main data, and which would allow for the fast cloning of a shared function table that you're looking for.

ValueMap would then look something like:

/// The Map value type used in Koto
#[derive(Clone, Debug, Default)]
pub struct ValueMap {
    data: Arc<RwLock<ValueMapContents>>,
    immutable_data: Option<Arc<RwLock<ValueHashMap>>>,
    external_data: Option<Arc<RwLock<dyn ExternalValue>>>,
};

…and ExternalValue and ExternalDataId could be removed as Value variants.

Something I like about this is that modules like could expose their features as immutable. It’s never really felt right that there’s nothing to stop a user from saying number.pi = -1 (but users could still extend core modules with their own functionality if they like).

…and I haven’t figured out a few details of the expected behaviour of map operations over a mix of mutable and immutable entries (like how map operations should apply), but I’d need to start playing with some examples to get a feel for the right direction.