Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

`libqaul` Service API #247

Merged
merged 5 commits into from Jun 28, 2019

Conversation

@spacekookie
Copy link
Member

commented Jun 26, 2019

Following is a draft PR for the libqaul service API. It is in no way representative of what the actual API will look like in the end, and is merely meant as a scope draft, i.e. what functions should be available, what type of interaction do we want to encourage, what data is available and how is data managed.

Looking at the code, this will become very obvious. There are several FIXME blocks in comments above functions, but apart from that, I wanted to write down some questions that I asked myself when writing this PR.

  • How to handle context? We can either have &self in functions, associated to a context struct, or we can have external API functions that are handed a reference to some context object. The latter is more C-friendly
  • How to represent users? Right now, the UserID is just a type alias to String. I assume we want to use some hash ID that was added by #242.
  • Sync vs Async? Right now the recv_hook is synchronous. I guess we might want to wait for the stabilisation of async/await before we add async API functions. But synchronous functions should still be available.
  • How to handle keys? Right now, the data storage is a single module and it should expose a generic data store. Do we want to have a separate API for key storage? Or is the keystore not something explicitly exposed, instead just using the User abstraction?
  • User creation builders? We could use a builder-style pattern for creating users with metadata. Alternatively, we could just have a single function that takes some object that is filled with data. The former is more Rust friendly, the latter more C friendly.
  • QResult wrapping internal Results? What is definitely a possibility for us to wrap internal errors that libqaul (and other dependencies such as RATMAN expose, then using failure to propagate up errors. The question is if a service creator might even care that much about it. Maybe we just want to expose a simple unified error for all libqaul service problems.
  • Registering services? This might require a general challenge-response approach similar to OAuth where a user on a libqaul "instance" accepts a new service into usage. Services should also only be register able for single users. Ultimately, the service will then become advertised as that user (i.e. a node on the network).
  • Service dependencies? Something currently not at all in scope are service dependencies. This means that a service (say a geocaching app) would be able to declare that it depends on another service (say messaging). This might even allow us to go as far as letting services be install-able via the qaul.net network, where users then broadcast binary requests, for dependeny services: i.e. having a geocaching app that depends on a GPS service, which is also external to libqaul

Okay so that's a lot of open questions. I'm curious to hear your thoughts and feedback. The actual qaul.net usability (messaging, file sharing, VoIP, ...) would then be implemented as services, via this API.

A note for Jess: this means that realistically the HTTP API interfaces need to be provided for the services that qaul.net exposes (messaging, file sharing, etc...), not this service API directly. (or rather, I don't think exposing the service API via HTTP is a very high priority, but could be done in the future).

@spacekookie spacekookie force-pushed the service-draft branch from f58a2a2 to df57527 Jun 26, 2019

@Jess3Jane

This comment has been minimized.

Copy link
Contributor

commented Jun 27, 2019

Sync vs Async i feel waiting for async/await to land is reasonable.

Registering Services? So if i'm understanding this correctly services would be authorized per user? i'm wondering if there's a use for a more fine grained permission system, also in the OAuth direction, where for example you could authorize an application to view your profile but not delete it. That might add too much complexity though.

HTTP API i'm presently structuring the api such that each service should be able to provide it's own section of it so i don't imagine this service api being exposed at all.

@spacekookie

This comment has been minimized.

Copy link
Member Author

commented Jun 27, 2019

@Jess3Jane

This comment has been minimized.

Copy link
Contributor

commented Jun 27, 2019

We can either have &self in functions, associated to a context struct, or we can have external API functions that are handed a reference to some context object

i'd prefer the former. Context structs are much more natural in rust and to a certain degree we'll need to build wrappers for the c api to deal with the regardless (for external structs like iron::Request) so we might as well have one language's api make sense.

We should sit down after this (or maybe at the same time), and talk about what kind of API the actual qaul.net services should have.

Sure, yeah. i'm presently working on the few layers of middleware that sit above all incoming requests but once I'm done with that that will be an excellent discussion to have.

Refactoring proposed service structure
- Moving `Error` to `qaul-common`
- Switching to `qaul-common` "identity" added by #242
- Shuffling around `qaul-common` internals
@spacekookie

This comment has been minimized.

Copy link
Member Author

commented Jun 27, 2019

On 19-06-27 06:43, Jess wrote:

is there any way to enumerate contacts or do you have to know a
contact exists to get their info?

Ah damn, yea I had completely missed that! There should be,
absolutely!

i'd prefer the former. Context structs are much more natural in rust
and to a certain degree we'll need to build wrappers for the c api
to deal with the regardless (for external structs like
iron::Request) so we might as well have one language's api make
sense.

I was leaning that way too. I'll add a context struct at the root of
libqaul in a second and then we can see how that feels from there. I
guess initialisation order of an application is something to consider
here.

An app initialises libqaul, then initialises various services, that
connect to libqaul. libqaul itself doesn't own the runtime, but
the app that calls it does.

Oh and @NoraCodes, I'm realising that some of the code for working
with services was already added by #244. If you don't mind I'll
cherry-pick that commit, then close the other PR.

Also something else to consider: We have multiple layers of objects
that are more or less the same thing, but we re-introduce them on
different levels because they are semantically different and we don't
want them coupled so tightly. Some of the repetition makes sense, some
doesn't. I prepared the table below and I'd love your feedback on what
you think.

User Message Payload
Service User Profile Signature verified message Text, File, ...
libqaul Fingerprint Signed message with payload Vec<u8> realistically
RATMAN Fingerprint Message with delivery chksum Vec<u8> realistically
NetMod IP, ... Low-level network frames Same as Message ...

We already discussed that in a message, the UserID for routing and
fingerprint are kinda the same and we want to re-use those. We could
make this part of the code too. There is ratman::identity as a
module which adds utilities around building a user ID from some hash
or binary data or whatever.

Next up, the messages: I think the biggest difference between a
message inside libqaul and also RATMAN (which again, could be
possibly shared struct wise) and a message handed to a service is,
that the service message should handle signatures differently.

Realistically the signature should be replaced with an enum like:

enum Signature
     Valid(UserID),      // Verified with this users pubkey
     Unverified(UserID), // Unverified message from a user
     Invalid,            // Failed validity checking: danger!
}

This way the cryptography is done inside libqaul, a service doesn't
have to explicitly call some auth module or make sure they
actually check the signatures. Sure, they can still work with
Signature::Invalid messages. But I think this would also be much
nicer as a service author.

At which point, the message layer is essentially the same from just
below the service API, to RATMAN splitting up a message into
multiple frames to shuv it into a network socket.

... anyway, sorry for rambling. Thoughts?

(reposted from the web-view because github emails don't allow markdown :( )

@NoraCodes

This comment has been minimized.

Copy link
Contributor

commented Jun 27, 2019

Oh and @NoraCodes, I'm realising that some of the code for working with services was already added by #244. If you don't mind I'll cherry-pick that commit, then close the other PR.

Yes, please do that.

We already discussed that in a message, the UserID for routing and
fingerprint are kinda the same and we want to re-use those. We could
make this part of the code too. There is ratman::identity as a
module which adds utilities around building a user ID from some hash
or binary data or whatever.

I think this is a good idea; unifying identity across as many levels as possible is certain to be beneficial to development speed and security.

@NoraCodes

This comment has been minimized.

Copy link
Contributor

commented Jun 27, 2019

My big question here is, do we expect each qaul.net application to have it's own set of registered services, or do we want the user to run some kind of qaul.net daemon onto which services register?

@Jess3Jane

This comment has been minimized.

Copy link
Contributor

commented Jun 27, 2019

a daemon seems best but may require management overhead that renders the software less accessible
this may be inevitable though

@NoraCodes

This comment has been minimized.

Copy link
Contributor

commented Jun 27, 2019

It would probably be doable to make nice installers that install the daemon along with whatever client software; I think that would be optimal. Also, if the daemon is sufficient to do routing, people could put it on headless SBCs and the like

@spacekookie

This comment has been minimized.

Copy link
Member Author

commented Jun 27, 2019

Create trait for connections to services
This can be used for "internal" services or those exposed over some kind
of network, or more exotic things. Service connectors that implement
only the syncronous version are very easy to implement, but there is an
extended trait that provides a polling method for async implementations.
@spacekookie

This comment has been minimized.

Copy link
Member Author

commented Jun 28, 2019

A change I kinda wanna make is removing the digest from Message. I think it would be reasonable to assume that digest checking is done by netmod, as such only Frame's have digest information. A Message is always assumed valid.

I kinda wanna merge this PR soon, just so it doesn't get way too big. @NoraCodes you could then just add the digest stuff you wrote for a message to the Frame abstraction instead (which I haven't done so far)

@spacekookie spacekookie force-pushed the service-draft branch from eda85e1 to cad4771 Jun 28, 2019

Restructuring `ratman` crate
Initially I was a little eager to split lots of components
into their own crates, to speed up compliation and add some
separation. But realistically, none of that is really
required. I removed the `routing-core` and `diagnostic` crates
and moved them to their own modules instead.

This way we don't run into problems with circular dependencies.

This commit also changes the way that `Frames` work. First it
re-adds the `trait` abstraction for an `Endpoint`, and also
removes the `broadcast` function, instead relying on `Frame`
to have a `None` `recipient`.

The semantic separation between a `Message` and a `Frame` is
that the latter has sequence numbers, and can thus compose
a Message as a series of smaller chunks.
How the signature should be handled in this case isn't really
clear for now. I guess we might want to replace the `signature`
on Frames alltogether and just have per-frame checksums to make
sure that no delivery errors were introduced.

@spacekookie spacekookie force-pushed the service-draft branch from cad4771 to 1692060 Jun 28, 2019

@spacekookie

This comment has been minimized.

Copy link
Member Author

commented Jun 28, 2019

Again, I've taken a very crate-happy approach in the beginning. I wasn't entirely sure how many cyclical dependencies we might end up with. Or what types needed to be exposed to what other
crates.
But I think most of that was pretty overkill. So I'm getting rid of the common crate, merging the types either into the ratman crate (see previous commit) or the api module.

The API is now using a struct and associated functions, all of them read-only. I sugest we use lots of internal mutability to make it all work.
Oh, I've also removed the data API for now. There were too many open questions about how to use it, we can always re-add one.

Anyway, I'll merge this PR after this then and we can move on from there.

@spacekookie spacekookie merged commit c99f4da into master Jun 28, 2019

1 check passed

ci/gitlab/service-draft Pipeline passed on GitLab
Details
@spacekookie spacekookie referenced this pull request Jun 29, 2019
spacekookie added a commit that referenced this pull request Aug 9, 2019
`libqaul` Service API (#247)
* Adding first draft of the service API

* Refactoring proposed service structure

- Moving `Error` to `qaul-common`
- Switching to `qaul-common` "identity" added by #242
- Shuffling around `qaul-common` internals

* Create trait for connections to services

This can be used for "internal" services or those exposed over some kind
of network, or more exotic things. Service connectors that implement
only the syncronous version are very easy to implement, but there is an
extended trait that provides a polling method for async implementations.

* Restructuring `ratman` crate

Initially I was a little eager to split lots of components
into their own crates, to speed up compliation and add some
separation. But realistically, none of that is really
required. I removed the `routing-core` and `diagnostic` crates
and moved them to their own modules instead.

This way we don't run into problems with circular dependencies.

This commit also changes the way that `Frames` work. First it
re-adds the `trait` abstraction for an `Endpoint`, and also
removes the `broadcast` function, instead relying on `Frame`
to have a `None` `recipient`.

The semantic separation between a `Message` and a `Frame` is
that the latter has sequence numbers, and can thus compose
a Message as a series of smaller chunks.
How the signature should be handled in this case isn't really
clear for now. I guess we might want to replace the `signature`
on Frames alltogether and just have per-frame checksums to make
sure that no delivery errors were introduced.

* Refactoring service API and `libqaul` internals
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.