libqaul Service API

[spacekookie]

Following is a draft PR for the libqaul service API. It is in no way representative of what the actual API will look like in the end, and is merely meant as a scope draft, i.e. what functions should be available, what type of interaction do we want to encourage, what data is available and how is data managed.

Looking at the code, this will become very obvious. There are several FIXME blocks in comments above functions, but apart from that, I wanted to write down some questions that I asked myself when writing this PR.

• How to handle context? We can either have &self in functions, associated to a context struct, or we can have external API functions that are handed a reference to some context object. The latter is more C-friendly
• How to represent users? Right now, the UserID is just a type alias to String. I assume we want to use some hash ID that was added by Building out the API #242.
• Sync vs Async? Right now the recv_hook is synchronous. I guess we might want to wait for the stabilisation of async/await before we add async API functions. But synchronous functions should still be available.
• How to handle keys? Right now, the data storage is a single module and it should expose a generic data store. Do we want to have a separate API for key storage? Or is the keystore not something explicitly exposed, instead just using the User abstraction?
• User creation builders? We could use a builder-style pattern for creating users with metadata. Alternatively, we could just have a single function that takes some object that is filled with data. The former is more Rust friendly, the latter more C friendly.
• QResult wrapping internal Results? What is definitely a possibility for us to wrap internal errors that libqaul (and other dependencies such as RATMAN expose, then using failure to propagate up errors. The question is if a service creator might even care that much about it. Maybe we just want to expose a simple unified error for all libqaul service problems.
• Registering services? This might require a general challenge-response approach similar to OAuth where a user on a libqaul "instance" accepts a new service into usage. Services should also only be register able for single users. Ultimately, the service will then become advertised as that user (i.e. a node on the network).
• Service dependencies? Something currently not at all in scope are service dependencies. This means that a service (say a geocaching app) would be able to declare that it depends on another service (say messaging). This might even allow us to go as far as letting services be install-able via the qaul.net network, where users then broadcast binary requests, for dependeny services: i.e. having a geocaching app that depends on a GPS service, which is also external to libqaul

Okay so that's a lot of open questions. I'm curious to hear your thoughts and feedback. The actual qaul.net usability (messaging, file sharing, VoIP, ...) would then be implemented as services, via this API.

A note for Jess: this means that realistically the HTTP API interfaces need to be provided for the services that qaul.net exposes (messaging, file sharing, etc...), not this service API directly. (or rather, I don't think exposing the service API via HTTP is a very high priority, but could be done in the future).

[Jess3Jane]

Sync vs Async i feel waiting for async/await to land is reasonable.

Registering Services? So if i'm understanding this correctly services would be authorized per user? i'm wondering if there's a use for a more fine grained permission system, also in the OAuth direction, where for example you could authorize an application to view your profile but not delete it. That might add too much complexity though.

HTTP API i'm presently structuring the api such that each service should be able to provide it's own section of it so i don't imagine this service api being exposed at all.

[spacekookie]

On 19-06-26 07:47, Jess wrote: **Registering Services?** So if i'm understanding this correctly services would be authorized per user? i'm wondering if there's a use for a more fine grained permission system, also in the OAuth direction, where for example you could authorize an application to view your profile but not delete it. That might add too much complexity though.

I actually kinda really like this idea. Especially when it comes to dealing with potentially malicious services, this could also prevent some damage. It's probably out of scope for now, but something we should definitely have on the roadmap!

**HTTP API** i'm presently structuring the api such that each service should be able to provide it's own section of it so i don't imagine this service api being exposed at all.

Cool! We should sit down after this (or maybe at the same time), and talk about what kind of API the actual `qaul.net` services should have. Obviously messaging is pretty simple, then there's file sharing (which is essentially messaging but with large attachments and stuff. And then VoIP (which isn't really over IP at all but eh 🤷).

[Jess3Jane]

We can either have &self in functions, associated to a context struct, or we can have external API functions that are handed a reference to some context object

i'd prefer the former. Context structs are much more natural in rust and to a certain degree we'll need to build wrappers for the c api to deal with the regardless (for external structs like iron::Request) so we might as well have one language's api make sense.

We should sit down after this (or maybe at the same time), and talk about what kind of API the actual qaul.net services should have.

Sure, yeah. i'm presently working on the few layers of middleware that sit above all incoming requests but once I'm done with that that will be an excellent discussion to have.

[spacekookie]

On 19-06-27 06:43, Jess wrote:

is there any way to enumerate contacts or do you have to know a
contact exists to get their info?

Ah damn, yea I had completely missed that! There should be,
absolutely!

i'd prefer the former. Context structs are much more natural in rust
and to a certain degree we'll need to build wrappers for the c api
to deal with the regardless (for external structs like
iron::Request) so we might as well have one language's api make
sense.

I was leaning that way too. I'll add a context struct at the root of
libqaul in a second and then we can see how that feels from there. I
guess initialisation order of an application is something to consider
here.

An app initialises libqaul, then initialises various services, that
connect to libqaul. libqaul itself doesn't own the runtime, but
the app that calls it does.

Oh and @NoraCodes, I'm realising that some of the code for working
with services was already added by #244. If you don't mind I'll
cherry-pick that commit, then close the other PR.

Also something else to consider: We have multiple layers of objects
that are more or less the same thing, but we re-introduce them on
different levels because they are semantically different and we don't
want them coupled so tightly. Some of the repetition makes sense, some
doesn't. I prepared the table below and I'd love your feedback on what
you think.

	User	Message	Payload
Service	User Profile	Signature verified message	Text, File, ...
libqaul	Fingerprint	Signed message with payload	Vec<u8> realistically
RATMAN	Fingerprint	Message with delivery chksum	Vec<u8> realistically
NetMod	IP, ...	Low-level network frames	Same as Message ...

We already discussed that in a message, the UserID for routing and
fingerprint are kinda the same and we want to re-use those. We could
make this part of the code too. There is ratman::identity as a
module which adds utilities around building a user ID from some hash
or binary data or whatever.

Next up, the messages: I think the biggest difference between a
message inside libqaul and also RATMAN (which again, could be
possibly shared struct wise) and a message handed to a service is,
that the service message should handle signatures differently.

Realistically the signature should be replaced with an enum like:

enum Signature
     Valid(UserID),      // Verified with this users pubkey
     Unverified(UserID), // Unverified message from a user
     Invalid,            // Failed validity checking: danger!
}

This way the cryptography is done inside libqaul, a service doesn't
have to explicitly call some auth module or make sure they
actually check the signatures. Sure, they can still work with
Signature::Invalid messages. But I think this would also be much
nicer as a service author.

At which point, the message layer is essentially the same from just
below the service API, to RATMAN splitting up a message into
multiple frames to shuv it into a network socket.

... anyway, sorry for rambling. Thoughts?

(reposted from the web-view because github emails don't allow markdown :( )

[NoraCodes]

Oh and @NoraCodes, I'm realising that some of the code for working with services was already added by #244. If you don't mind I'll cherry-pick that commit, then close the other PR.

Yes, please do that.

We already discussed that in a message, the UserID for routing and
fingerprint are kinda the same and we want to re-use those. We could
make this part of the code too. There is ratman::identity as a
module which adds utilities around building a user ID from some hash
or binary data or whatever.

I think this is a good idea; unifying identity across as many levels as possible is certain to be beneficial to development speed and security.

[NoraCodes]

My big question here is, do we expect each qaul.net application to have it's own set of registered services, or do we want the user to run some kind of qaul.net daemon onto which services register?

[Jess3Jane]

a daemon seems best but may require management overhead that renders the software less accessible
this may be inevitable though

[NoraCodes]

It would probably be doable to make nice installers that install the daemon along with whatever client software; I think that would be optimal. Also, if the daemon is sufficient to do routing, people could put it on headless SBCs and the like

[spacekookie]

On 19-06-27 10:59, Leonora Tindall wrote: My big question here is, do we expect each qaul.net application to have it's own set of registered services, or do we want the user to run some kind of qaul.net daemon onto which services register?

This is something that @luxferresum and I had a conversation about and figured out a cool idea: every service that wants to use a `qaul` network is gonna bundle a copy of `libqaul` anyway, right? And there's probably already a platform specific wrapper involved, like an android shim or something that creates a linux daemon or whatever. Then in this shim we can do one of two things: 1. If we are the only/only qaul service running on the device, we take ownership of the network. This is essentially what `qaul.net` will do. 2. If there is another service running on the system, we do some $IPC_MAGIC to connect to it, so that it can then re-use the registered users, etc. On Android this can be done via intents, on Linux via checking if some service is running (or checking some socket/env var/whatever.

[spacekookie]

A change I kinda wanna make is removing the digest from Message. I think it would be reasonable to assume that digest checking is done by netmod, as such only Frame's have digest information. A Message is always assumed valid.

I kinda wanna merge this PR soon, just so it doesn't get way too big. @NoraCodes you could then just add the digest stuff you wrote for a message to the Frame abstraction instead (which I haven't done so far)

[spacekookie]

Again, I've taken a very crate-happy approach in the beginning. I wasn't entirely sure how many cyclical dependencies we might end up with. Or what types needed to be exposed to what other
crates.
But I think most of that was pretty overkill. So I'm getting rid of the common crate, merging the types either into the ratman crate (see previous commit) or the api module.

The API is now using a struct and associated functions, all of them read-only. I sugest we use lots of internal mutability to make it all work.
Oh, I've also removed the data API for now. There were too many open questions about how to use it, we can always re-add one.

Anyway, I'll merge this PR after this then and we can move on from there.