Swift platform

[mcalligator]

m-ld for Swift is coming soon (we're already working on it). This is for peer-to-peer live data sharing among iOS, iPadOS and MacOS devices (and with other platforms that can run m-ld, such as JavaScript).

Please +1 this issue to vote for it.

[kkostov]

Is this effort open to contributions? I would be very happy to help with the shaping and implementation of the Swift components.

[gsvarovsky]

Hi @kkostov 👋, very open, thanks for the offer!

Our top priority on the Swift platform is to create a great dev experience. So we're looking at having a very "Swifty" API that's object-oriented and reactive (via Compose), which wraps around the core m-ld API (three methods and lots of JSON 🤖).

Here's our thinking so far:

https://github.com/m-ld/m-ld-swift-poc

We'd be really interested in your thoughts about that... and if you've got other ideas we'd love to talk about those too!

[kkostov]

Awesome, sounds great! I will have a look at the POC and share some insights.

[mcalligator]

Hey @kkostov, great to have you stop by! Thanks for your offer to contribute; would be fantastic to get the input of someone with a lot of relevant experience to help make this as good as it can possibly be. Looking forward to hearing your feedback on the PoC, and to working with you!

[kkostov]

My apologies for the late reply. It's been one of those weeks 😅.

I had a chance to review the PoC, and you did a great job translating a complex concept into an understandable client library! 🙌 I was delighted to see Swift Concurrency features like actors in use for the OrmDomain implementation. I think this technique is a perfect fit.

I would like to understand your concrete goals for the PoC, e.g., evolving the PoC to become a stand-alone library. Are there technological/conceptual aspects that remain experimental or theoretical and need to be de-risked within the scope of the PoC?

The JavaScript library can be an inspiration here (although perhaps Meld for iOS can offer an even higher level API to "flatten" the initial learning curve). For example, consider making an SPM package that implements an OrmDomain and various storage and sync/transport options a developer can choose e.g.

Meld() // in memory storage with basic sync support

Meld(domain: .json) // local json storage with basic sync support

Meld(domain: MyCustomDomain()))

Meld(sync: MeldMultiPeerSync())

I like the idea of being able to select subjects based on a query or some predicate (do you see this as a custom DSL in the future?). Consider, for example, the implementation details of AllConferences abstracted away behind an attribute (e.g., @query(MyQuery())) which takes a developer-provided argument for the selection (e.g., a type, an instance of a "request," graphQL query, etc..). CoreData and other SQLite libraries have a similar approach, e.g., https://github.com/groue/GRDB.swift#the-query-interface.

I also noticed that Subjects are instances of an object (reference types). It will be great if the choice between value and reference types remains with the developer.

What is a good first step that I could undertake?

[gsvarovsky]

Hey Konstantin! Thanks for your review and for your thoughts & ideas, wonderful 🙏

Our goals for the PoC are:

1. It makes sense to a Swift-native engineer. Sounds like that's a ✅
2. Better API than Core Data & SQLite for usability, as well as for sync & semantics
3. Incrementally and transparently (to 2.) work up to full m-ld protocol compliance

So the next steps include upwards (API) improvements, and downward (core) implementation. Lots more details below. My intuition is that you might be more interested with the API side. In any case, I think the first concrete step is to expand the example, so that it tests out writing data to the domain (and so, shakes out the usability of ideas like commit() and the deleted flag).

Although, if you think it's better to start with system tests rather than an example app, that's cool too. Any other ideas you have are most welcome. Maybe we can keep looking at the bigger picture in this thread, at the same time.

API

The constraints on the API are basically just that it must be implementable with m-ld... though of course we always want to improve m-ld, and we certainly will based on this work!

The PoC is definitely a higher-level API, for flattening the learning curve, in the way you suggest. In that role it's opinionated, for example about references. Note that we've been working on a higher-level object-oriented API in Javascript too. I'm writing baseline documentation for it right now, and I'll link that here very soon.

Yes, the PoC should evolve into a library, increasing the separation between it and the example app. I think one SPM package for the main lib is probably the way to go for now, but maybe other API patterns ultimately deserve their own lib. For example, one possible future packaging might be:

graph RL
  C(Functional API) --> A(m-ld Core)
  B --> A
  subgraph "(Current POC)"
    D[Conferences Example App] --> B(Object-Oriented API)
  end

Loading

With regard to separating storage and transport options, that's how the JS engine works, with its entry point, which uses injected dependencies for each. Having swappable sync options (concurrency models) too – that's very interesting. The idea with m-ld is that it will be able to support multiple concurrency models (concurrently!). That's the direction of some of our research right now. Having this choice on initialisation is certainly something we could do, even if we change how the choice is made in future.

I love @Query(MyQuery()))! I was a bit uncertain about how to library-ify AllConferences and this sounds great. We do already have an answer for developer-provided queries: our DSL is json-rql. In fact, all the JSON in the m-ld core API is json-rql, because it's a superset of JSON-LD. What we need, then, is a Swift-idiomatic way to construct json-rql. I did it in Java using a builder pattern (note that here we won't need translation into SPARQL, because m-ld itself is a json-rql native).

Other key open research questions in the API design are:

• Lazy loading. Is it needed, and how to represent it? (e.g. like "faults" in CoreData.)
• Compositions. How do we mark an object property as a composition, so that deletes are cascaded?
• I'm not too happy with the current way that you can make changes to OrmSubjects anytime, but if an OrmState is active the changes aren't shared until you call commit(). Seems a bit too magic and something a new dev could trip over. Maybe just always require a commit...

core

So, how are we going to implement the API, whatever shape it takes? The core m-ld protocol is still in motion, so our idea here is to learn as much as we can with a simulated engine before diving into a full implementation. However, the simulation itself should be as useful as possible right from the start!

We can learn a lot with a local un-synced JSON-file backend, but we should move as fast as possible to something with some sync, to shake out how the API behaves with concurrent changes.

So one idea is to implement a back-end sync service, deployable locally with Docker and also on the cloud. This service would use m-ld-js internally and expose the clone JSON API via HTTP. Since this introduces the possibility of concurrent clients, we'd implement some naive optimistic locking just to ensure data integrity. So some transaction commits may incur a lock violation; which you wouldn't see if you used a local m-ld engine.

Another option is to try and package up the m-ld-js engine, so it can be run as a process locally, in parallel with the app that's using the Swift API. Not being an iOS expert I don't know whether this is more trouble than its worth, or even possible! E.g. Interprocess communication on iOS with Mach messages sounds a bit arcane & brittle 😳. Plus, local persistence and network connections from the process might be tricky/impossible.

[kkostov]

Sounds good!

My intuition is that you might be more interested with the API side. In any case, I think the first concrete step is to expand the example, so that it tests out writing data to the domain (and so, shakes out the usability of ideas like commit() and the deleted flag)

Your intuition is correct. I can try implementing the commit and delete as the first PR. Speaking of tests, I will also try to add a few to the example app :)! It will help me better grasp how the framework works without having to run the entire app every time.

love @query(MyQuery()))! I was a bit uncertain about how to library-ify AllConferences and this sounds great. We do already have an answer for developer-provided queries: our DSL is json-rql. In fact, all the JSON in the m-ld core API is json-rql, because it's a superset of JSON-LD. What we need, then, is a Swift-idiomatic way to construct json-rql. I did it in Java using a builder pattern (note that here we won't need translation into SPARQL, because m-ld itself is a json-rql native).

Cool! There are not a lot of JSON-LD libraries currently available for Swift. To create a json-rql DSL, we can take advantage of some Swift 5.4+ features like result builders! We can start this as a separate SPM (which is not specific to Meld, but rather json-rql in general).

Lazy loading. Is it needed, and how to represent it? (e.g. like "faults" in CoreData.)

The answer is a solid Yes. App performance is paramount when building apps for iOS. As a developer, I need a way to quickly access a subset of data with minimal impact on the system. One example is the expectation that a native app should be loaded and interactive by the time the launch animation is complete (that is, the animation that "expands" the app from the icon when you tap it on the Home screen). That's under 400ms to load the start screen, get data and render it.
High-performance access is also preferred in cases like collection views (UIKit) or Lists (SwiftUI). Ideally, a developer should avoid implementing or even discovering faults.

How to represent it is an interesting question. My first thought is that it should be transparent, e.g., the library does what it needs to do by default. Some options to customize this can be provided, e.g., to force a query to be loaded on demand.

Compositions. How do we mark an object property as a composition, so that deletes are cascaded?

Good question. If a relation is defined using a property attribute, perhaps this can be added as an argument, e.g.:

class Notebook {
// ...
 var notes: Note[]
}

class Note {
// ...
 @MeldField("notebookId", onDelete: .cascade)
 var notebook: Notebook
 var notebookId: String
}

I'm not too happy with the current way that you can make changes to OrmSubjects anytime, but if an OrmState is active the changes aren't shared until you call commit(). Seems a bit too magic and something a new dev could trip over. Maybe just always require a commit

Agreed, I think having a clear action to designate saving/updating changes makes a lot of sense. I'd also consider a .rollback() or similar which resets the subject back to last known state (e.g. in case of accidental writes on the objects).

So one idea is to implement a back-end sync service, deployable locally with Docker and also on the cloud.

That's a good idea. For simplicity, it could use the already existing js library (for a Node-based solution), or we could start creating a Swift Package, which can be used both on the client and the server e.g using Vapor.

[gsvarovsky]

Great thoughts & actions, thanks Konstantin!

I can try implementing the commit and delete as the first PR. Speaking of tests, I will also try to add a few to the example app :)!

Awesome, thanks. You could create a ticket in the repo for further discussion if you like. In any case, I've given you direct write access to the repo.

To create a json-rql DSL, we can take advantage of some Swift 5.4+ features like result builders! We can start this as a separate SPM (which is not specific to Meld, but rather json-rql in general).

Agreed. Would you like to create a repo for that? – you'll have a much better idea that me about how best to set that up. (I've generally created json-rql stuff on my personal account.)

I've created three tickets in the POC repo for these bits:

• Lazy loading m-ld-swift-poc#1
• Support for Compositions m-ld-swift-poc#2
• Explicit transaction boundaries m-ld-swift-poc#3

For simplicity, it could use the already existing js library (for a Node-based solution), or we could start creating a Swift Package, which can be used both on the client and the server e.g using Vapor.

Yes! The Swift Package is the actual Swift engine (this ticket!). We can certainly embark on that. It would help for us to first document the clone protocol. We've held off on that so far because it's still in flux.

Packaging up the Node library as a process/image in the interim has also just come up in another discussion, in relation to using it in a PHP environment (before having a PHP engine). I'll reference both motivations from here into #41.