How do you communicate with the outer world from Rid ?

[Roms1383]

Okay, I've made a few progress in the meantime, and it would be very long to detail everything but basically :

• rid cannot be used if diesel or rocket exists in the dependencies (I personally had to do A LOT of conditional compile and feature flags, since my workspace aims at multiple facets)

  update: well not exactly, it doesn't work with diesel feature postgres which was brought by one of my dependencies, but it indeed does work with diesel feature sqlite, see my comment below

• enum cannot be used in rid just yet

update: I was wrong on this, see my comment below

• as stated in this other issue, Option<T> is not supported in rid just yet

---

Once I overcame all these previous issues I now have some script that I call inside update, like so:

impl RidStore<Msg> for Store {
    // ...
    fn update(&mut self, req_id: u64, msg: Msg) {
        match msg {
            Msg::Sync => {
                let handle = tokio::runtime::Handle::current();
                let _ = sync(handle, req_id);
            }
        }
    }
}
fn sync(handle: Handle, req_id: u64) -> Result<(), Box<dyn Error + Send + Sync>> {
    handle.spawn(async move {
        // ... request remote grpc server and stream responses (e.g. server-side streaming)
        // ... until stream reaches the end
        rid::post(Reply::Synced(req_id));
    })
}

And as you probably already guessed, I'm hitting a wall here since I can't find how to start a tokio runtime in Rid context, nor can I store it in Store itself since it results in: Cannot convert unknown rust type to dart type.

I went to search for a macro attribute like e.g. #[rid::skip] or #[rid::ignore] but couldn't find one.

My guess is that I could find a trick to create the runtime inside update and use some boolean flag to avoid recreating it every time, but I haven't tried yet and it feels super hacky.

---

Which brings another related question:
provided I manage to run tokio runtime and call grpc just fine, then how do I even store the results locally ?

I might have missed something but the fact that having diesel in the dependencies makes the build fails doesn't give me much faith that I can use a local storage of any kind on the device (I naively thought that I could go with SQLite).

update: I was wrong on this, see my comment below

So what are the options left ?
Do I need to go through calling Dart and leave it to Dart to save the results with e.g. Hive for example ?

update: I was wrong on this, see my comment below

IMHO that would just defeat the purpose of using Rust in the first place 😐 or maybe I didn't understood Rid use-cases ? 🤔

update: I could utterly have misunderstood the correct usage of Rid, but at least SQLite should be usable, see my comment below

[Roms1383]

Update! Ok my bad regarding diesel I made a subsequent test with diesel = { version = "1.4.6", features = ["sqlite"] } and indeed this compiles and runs. Good news !

[Roms1383]

Well maybe I panicked for nothing 😅

Seems like I can use tokio by manually building it and managing it, still some errors coming but I'll keep you updated.

[Roms1383]

Ok everything finally worked, here are a few notes if you need async grpc remote calls:

• tokio Builder needs enable_io inside rid plugin
• app needs the entitlements for client on macos as specified here
  My bad and apologies for the long mono thread, I'll close this now.

[Roms1383]

Oh well I'm re-browsing the docs and indeed I owe another apology: enum are indeed supported.

[SecondFlight]

Hi @Roms1383, would you be willing to post an example of how you're setting up and managing your tokio runtime? I think it might save me a lot of time 😅

[Roms1383]

Oh sure @SecondFlight right now it's still pretty rudimentary but here it is:

impl RidStore<Msg> for Store {
    // ...
    fn update(&mut self, req_id: u64, msg: Msg) {
        match msg {
            Msg::Sync => {
                // TODO: how to know if tokio runtime is recreated on each calls ?
                // investigate further ...
                rid::post(Reply::Syncing(req_id));
                println!("hello Tokio");
                let runtime = tokio::runtime::Builder::new_multi_thread()
                     // this is required to make remote grpc calls with tonic client (also don't forget ios entitlements in settings)
                    .enable_io()
                    .build()
                    .expect("runtime");
                runtime.block_on(async {
                    println!("inside block on");
                    let uri = "https://127.0.0.1:50051".parse::<Uri>().expect("valid uri");
                    // just a helper method to connect grpc client
                    match protocol::service::connect(uri).await {
                        Ok(client) => {
                            // here it's a stream because I use grpc server-side streaming
                            let mut stream = client.sync().await.expect("stream");
                            loop {
                                match stream.message().await {
                                    // type received over the wire: generated by tonic-build
                                    Ok(Some(protocol::domain::MyTypeProtobuf {
                                        items,
                                    })) => {
                                        eprintln!("items = {:?}", items);
                                        // local type that allows for sending over FFI
                                        // (identical to type generated by tonic-build, but with FFI-compliant typed fields and a TryFrom impl, see below)
                                        let mut mapped: Vec<MyTypeFFI> = vec![];
                                        for item in items {
                                            mapped.push(MyTypeFFI::try_from(item).expect("invalid data"));
                                        }
                                        eprintln!("mapped = {:?}", mapped);
                                        self.items.append(&mut mapped);
                                    }
                                    Ok(None) => {
                                        eprintln!("finished");
                                        break;
                                    }
                                    Err(e) => {
                                        eprintln!("{:#?} stream failure", e);
                                        rid::post(Reply::SyncFailure(req_id));
                                        break;
                                    }
                                }
                            }
                            eprintln!("synced");
                            rid::post(Reply::Synced(req_id));
                        }
                        Err(e) => {
                            eprintln!("{:#?} connection error", e);
                            rid::post(Reply::SyncFailure(req_id));
                        }
                    };
                })
            }
        }
    }
}

[Roms1383]

As you can see I'm not yet managing the runtime at all. I'm still stuck on another minor thing: I try to use the rid plugin crate in different context and make cbindgen work with cargo features.

[SecondFlight]

Thanks so much! I'll put a pin in this for later.

[Roms1383]

@SecondFlight take what follows with a bit of salt, I might be wrong 😅
but I think that using try_current could be enough to manage tokio runtime, something like:

let runtime = tokio::runtime::Handle::try_current();
let runtime = if runtime.is_ok() { runtime.unwrap() } else {
 tokio::runtime::Builder::new_multi_thread()
    // this is required to make remote grpc calls with tonic client (also don't forget ios entitlements in settings)
    .enable_io()
    .build()
    .expect("runtime")
};

[Roms1383]

Yeah trying right now and I'm wrong because they return different types Runtime and Handle. Sorry I'll check it in the coming days and keep you updated.

[Roms1383]

Well just tested some more but runtime gets recreated on each call anyway.

[SecondFlight]

Ok, no worries! I probably won't get to this by then, but if I do I'll let you know what I find.

[thlorenz]

Looks like you both already went pretty deep here.
I never considered using rid with tokio as Flutter already provides an event loop.

Please LMK how I can help and yes you're right simple enums are supported (i.e. ones with just scalar values like the below).

#[rid::model]
pub enum Color {
  Green,
  Blue
}

They need to be identified where used via #[rid::enums], i.e.:

#[rid::model]
#[rid::enums(Color)]
struct MyStruct {
  color: Color
}

I need to improve/update the docs, unless one of you wants to fill in some examples for now in this section.
Given the lack of documentation, looking at the integration tests is a good way to see what is supported at this point and how.

Given this issue is closed maybe open another one if you run into more problems/need help.

[Roms1383]

Speaking about which, a cool and quick experiment that could be worth checking: start either a tokio, an async_std or a smol runtime on rid startup.
If it does indeed work, then gate it behind feature flags.
What do you think @SecondFlight, @thlorenz ?

[SecondFlight]

I'm sure that would work. With my limited understanding, I feel it should be possible to do this without baking it into Rid though... If it's relatively easy to do this, then it might be better to add documentation for how to do it instead. That way the library consumer can choose which one they want to use, and Rid won't need to support anything extra.

[thlorenz]

You should create a prototype of an app with those features @Roms1383 .
We can then have a look and see what makes sense to include with rid, even though I agree with @SecondFlight that most likely it shouldn't be included, but an example app with detailed documentation would serve others trying something similar.

I also do think adding a second event loop (given Flutter provides one already) is somewhat of a special case.

[Roms1383]

@thlorenz You should create a prototype of an app with those features @Roms1383 .

Ok I have quite a lot of things in the pipeline at the moment, but I'll try to do it before Christmas.
I also have the Rid CLI and a bunch of other stuff around that I'm supposed to carry on too 🙏 😅

Only thing I haven't look at yet is: how do you start rid ? Usually tokio runtime is started in fn main, either by annotating it with a macro attribute (#[tokio::main]) or by explicitly building it programmatically. I have to look into that.

The problem doing as in #57 (comment), is that I have to build a runtime on each call to update. Usually the runtime is built on startup, and then just retrieve a handle later like e.g.:

use tokio::runtime::Handle;

// Inside an async block or function.
let handle = Handle::current(); // or Handle::try_current(); in fallible situations
handle.spawn(async {
    println!("do some task asynchronously");
});

---

We can then have a look and see what makes sense to include with rid

Well to allow users to be able to use async / await syntactic sugar all around, then it's more convenient this way AFAIK 🤔.

---

I also do think adding a second event loop (given Flutter provides one already) is somewhat of a special case.

Not so special actually: library consumers who create a Rid plugin might also want to make remote calls (REST/gRPC/etc) or any kind of asynchronous tasks from Rust directly, instead of having to rely on Dart for these.This is even better in terms of decoupling since you can delegate all your client-side business logic to Rust.

---

@SecondFlight With my limited understanding, I feel it should be possible to do this without baking it into Rid though...

This is exactly what cargo features provides: conditional compilation.

In other term, library consumers who have no need for a runtime have nothing to do e.g. rid = "1.0", and those who wants it can add e.g. rid = { version = "1.0", features = ["tokio"] }. The related code and dependency doesn't even get included in the final binary if the feature is not activated, so pretty cool ✨

[Roms1383]

Would you mind activating the Discussions feature from your Github repository @thlorenz ?
I have the feeling that we should discuss over there instead of spamming your Github issues 😅

[thlorenz]

Hi @Roms1383 I like the idea of discussions, but as you can see I'm having a hard time keeping up with responding to issues/PRs as it is.
I'd prefer to keep this in issues, but we can discuss away in there to our hearts content. issue maybe a bad name as it's not always something bad or broken. Instead they can also be discussion threads for ideas.
Also a PR in draft mode is a great place to discuss ideas as the implementation can grow as part of the discussions and it is easier to comment on code samples.

Therefore what I'd suggest you do is the following. When you're ready to work/discuss a new feature/idea please open an issue or PR (whichever works best for the case). Then provide a detailed summary of what you're suggesting/attempting to implement and we will keep it open as long as we discuss the idea in there.
Important is to have one issue and/or PR per feature/idea. Feel free to open as many as you want. That is better than mixing discussions in one thread which gets hard to follow.

Thanks!

[Roms1383]

PR in draft mode indeed looks great, I didn't know about this feature !

[Roms1383]

As a quick follow-up, this evening I was browsing rid's files but I couldn't find anything looking close to a classic fn main function. My initial thought was to look at how rid could have a tokio::runtime running on startup till shutdown, to keep spawning tasks on it instead of recreating it every time as stated earlier in the comments (e.g. this one).

Is my understanding correct @thlorenz that rid is initialized as a flutter plugin, hence there's no main as in usual binaries ? Is my assumption correct that all the lift-up is done in e.g.:

import Flutter
import UIKit

public class RidPlugin: NSObject, FlutterPlugin {
  public static func register(with registrar: FlutterPluginRegistrar) {
    // ...
  }

  public func handle(_ call: FlutterMethodCall, result: @escaping FlutterResult) {
    result(nil)
  }

  public func dummyMethodToEnforceBundling() {
    // dummy calls to prevent tree shaking
    // ...
  }
}

I would assume that one can probably use some initialization function like e.g.:

#[no_mangle]
pub extern "C" fn initialize_runtime(...) -> ? {
  // build tokio runtime ...
}

And if that's the case I currently wonder how to keep the runtime around (not dropped) between FFI calls and I still have no clue yet.

[SecondFlight]

After a bit more thinking I believe @thlorenz is right, in that a separate async runtime in Rust is unnecessary for most use-cases. Flutter already provides an async runtime. Additionally, the way Rid is designed makes it safe to block in Rust code as doing so will not block the UI (with one caveat, see below). See the Reddit ticker app for an example. Note that the example uses ureq, which is single-threaded and blocks the active thread while waiting on a response (source).

It should also be fairly straightforward to use a two-way channel like a socket. Based on this snippet from the Reddit ticker app, I believe it's safe to call rid::post() from a separate thread. So you could do something like this:

1. Spawn a thread for the socket connection
2. Use a channel to forward requests from the main app to the request thread. Aside from possible ownership issues, this channel could be stored in the store and ignored via #[rid(skip)]
3. When responses are received, use rid::post() within that thread to notify the UI
4. Or, if the reply needs to trigger a state change, lock the store and mutate as shown in the example above, and send the relevant update messages via rid::post().

@thlorenz I have some questions about how Rid handles messages:

1. Is rid::reply() guaranteed to be thread-safe?
2. In the high-level API, if multiple messages are sent before a reply is received, what happens? I expect that Rid would only allow one message to be processed at once, but I want to double-check this assumption as it has safety implications.

---

Caveat to the first paragraph: if my assumptions are correct, a blocked message would prevent additional messages from being processed, which could have detrimental effects on usability.

[Roms1383]

Well I actually think it's really a viable case @SecondFlight in order to delegate all the business logic to Rust / rid side and using Flutter solely for UI / layout / animations / display. If I have to query the backend from Dart to then pass the results to the rid store, then somehow to me it sounds a bit like an architecture design "smell" in a way that it prevents from clearly decoupling the business logic from the display. That being said, most of async code has its sync counterpart, so there's always a way and also thanks for your suggestions regarding channels, it makes sense, I'll look into it in the coming days !

[SecondFlight]

I definitely won't argue that. What works best really depends on your architecture, use-cases, preferences, etc. My big a-ha moment in the last week is just that the whole stack is fundamentally async-capable without adding tokio.

[Roms1383]

Yes, utterly agree on this point too.