[Proof of concept]: Neos UI API

[grebaldi]

Hi there,

so, what's this all about, then?

I've been thinking about the Neos UI API for quite a while now and although I always had lots of ideas, I was never able to adequately put them into writing. So, I've decided to put my ideas to the test and just implement them straight alongside the existing UI code and share my findings with everyone.

This PR is not meant for merging, but is only here to showcase some ideas I have been thinking about. I hope that it will inspire discussion about those ideas, while I do not presume that everyone will agree with them.

TLDR; I've re-implemented part of the Inspector based on a new API design. On the server side, this design allows to model Query- and Command-objects. Code generation is then used to automatically create typesafe client code. The API also has a notion of asynchronous notifications.

This PR will remain a draft and is not meant to be merged.

In the following, I will try to describe the contents of this PR as good as I can. There's a lot of ground to cover, but I hope that this presentation will provide some understanding about the big picture. Any comments, questions, ideas and the like are very welcome :)

Design Goals

There are three major design goals underlying the concept demonstrated in this PR:

Move responsibility from the client to the server

At the moment, the Neos UI implements a lot of Neos domain concepts redundantly. This was done initially to avoid as many HTTP requests as possible, so the UI would always feel snappy. Especially with regard to the upcoming Event-Sourced Content Repository, this comes at a great cost, because almost every domain concept has to be thought about twice.

As of today, Neos shows excellent performance improvements that indeed allow to rethink this initial assumption. It is very likely feasible to move all the domain concerns back to the server at the cost of more HTTP Request exchange. This would be very good news for the UI client code base, because under those circumstances it could fully focus on frontend concerns.

End-to-end type safety

In terms of type safety, both the PHP and the JavaScript world have changed a lot since the beginning of the UI development. PHP's type system has become more versatile and JavaScript has almost been replaced by its structurally type-safe superset Typescript.

Typescript has already been introduced to the UI code base, but the current Backend API is hard to wrap into type definitions. This process also requires redundant code.

It should be possible, to implement Neos UI API objects in a way that allows us to leverage the type-safety of both worlds, without the need to write redundant code manually.

Extensibility

Of course, for the sake of plugin development, it should be possible to extend the Neos UI API with custom concepts. Ideally leveraging the advantages of the other two design goals.

The (server-side) API framework

This PR contains an entire framework for the API design I was thinking of. A separate package for these mechanisms is certainly warranted.

The API Framework establishes invariants as well as JSON-serialization/-deserialization procedures for the following primitives:

DTO

A DTO (Data Transfer Object) is a plain PHP object that follows a set of specific rules:

• It must extend the provided Dto class
• It must have a public constructor
• It must declare all its public properties as promoted, readonly constructor parameters
• All its public properties must have one of the following types:
  • string, int, float, bool
  • mixed
  • Class that extends Dto
  • Class that extends ListDto
  • A union of any of the above
• It must be immutable

DTOs are used to model arbitrary, serializable data structures for exchange between client and server.

• Example: https://github.com/neos/neos-ui/blob/77d1d176deaa6736733fb186cad50cd7e810a8f8/Classes/Application/Dto/Property/PropertyDto.php

ListDTO

A ListDTO represents collections of DTOs. It is itself a plain PHP object that follows a set of specific rules:

• It must extend the provided ListDto class
• It must have a public constructor
• It must have exactly one, variadic constructor parameter $items
• The constructor parameter $items must have one of the following types
  • string, int, float, bool
  • mixed
  • Class that extends Dto
  • A union of any of the above

Every ListDTO serializes to a plain array.

• Example: https://github.com/neos/neos-ui/blob/77d1d176deaa6736733fb186cad50cd7e810a8f8/Classes/Application/Dto/Property/PropertiesDto.php

Command & CommandHandler

A Command is a DTO that contains an instruction for performing a change through the application. It is handled by a corresponding CommandHandler and throws an Exception if anything goes wrong.

CommandHandlers are allowed to perform side-effects.

• Example (Command): https://github.com/neos/neos-ui/blob/77d1d176deaa6736733fb186cad50cd7e810a8f8/Classes/Application/Command/ChangeNodeProperties/ChangeNodePropertiesCommand.php
• Example (CommandHandler): https://github.com/neos/neos-ui/blob/77d1d176deaa6736733fb186cad50cd7e810a8f8/Classes/Application/Command/ChangeNodeProperties/ChangeNodePropertiesCommandHandler.php

Query, QueryHandler & QueryResult

A Query is a DTO that represents a question that the application can be asked. It is handled by a corresponding QueryHandler, which produces a QueryResult and throws an Exception if anything goes wrong.

A QueryHandler should not perform side-effects.

A QueryResult is also a DTO.

• Example (Query): https://github.com/neos/neos-ui/blob/77d1d176deaa6736733fb186cad50cd7e810a8f8/Classes/Application/Query/GetInspector/GetInspectorQuery.php
• Example (QueryHandler): https://github.com/neos/neos-ui/blob/77d1d176deaa6736733fb186cad50cd7e810a8f8/Classes/Application/Query/GetInspector/GetInspectorQueryHandler.php
• Example (QueryResult): https://github.com/neos/neos-ui/blob/77d1d176deaa6736733fb186cad50cd7e810a8f8/Classes/Application/Query/GetInspector/GetInspectorQueryResult.php

Inbox

The Inbox concept exists to enable asynchronous interaction between server and client. As of right now, every user has their own Inbox. I noticed during implementation that this doesn't make much sense and Inboxes should rather be bound to a Workspace (CR). For demonstration purposes however, I left it with the user-binding.

Also, in this PR, Inboxes are backed with a SQLite3 implementation, which is great for demo purposes, because it doesn't require a database migration, but would be very unwise for production. It's however possible to handle Inboxes with the regular persistence mechanism of Neos (Redis would be even better, though)

On the client side, Inboxes are polled every 10 seconds. This is a lot of time, but for actual asynchronous events, it is okay. To remedy the effects of this long interval, Inboxes are also polled immediately after a command was executed (thanks to @mficzel for the idea 🙂).

Inboxes contain two kinds of objects:

Notification

A Notification is a DTO that contains information about an event that happened outside the usual Request/Response cycle of the application. Notifications can be sent to one or more Inboxes.

Example: https://github.com/neos/neos-ui/blob/77d1d176deaa6736733fb186cad50cd7e810a8f8/Classes/Application/Notification/PropertiesWereUpdatedNotification.php

Status

A Status is a DTO that contains state information that can be actively pushed from the server to the client outside the usual Request/Response cycle of the application. Statuses can be updated for one or more Inboxes.

This concept exists in the framework, but lacks an example. The idea was to enable things like: "Is somebdoy else editing the document I'm looking at right now?".

The Client-side framework

@neos-project/framework-schema

The @neos-project/framework-schema package provides functions for defining a data schema that can be used to validate any incoming data strucure.

For example:

import * as s from "@neos-project/framework-schema";

const PropertyDto = s.object({
    name: s.optional(s.string()),
    value: s.union(s.string(), s.number()),
});

PropertyDto.is({ value: 42 }); // will be false, because "name" is missing
PropertyDto.accepts({ value: 42 }); // will be true, because "name" can be added as `null`

const myProperty = PropertyDto.ensure({ value: 42 }); // `myProperty` will contain `{ name: null, value: 42 }` and will have the proper typescript type
const myOtherProperty = PropertyDto.ensure({ 
  name: ["oops an array instead of a string"], 
  value: 42, 
}); // This will throw an Error

The Infer utility type can be used to derive a proper TypeScript type from a defined schema, thus achieving both, static and runtime type safety.

By using the Infer utility, the schema from above can be turned into a type:

type PropertyDto = s.Infer<typeof PropertyDto>;

Which corresponds to:

type PropertyDto = {
  name: null|string;
  value: string|number;
}

This package contains measures to deal with json_encode (PHP) output properly. For instance, the s.hashMap() schema will accept both empty objects and empty arrays:

const MyDto = s.hashMap(s.object({ baz: s.number() }));

MyDto.is({}); // true
MyDto.is({ 
  foo: { 
    baz: 42 
  },
  bar: { 
    baz: 84
  },
}); // true
MyDto.is([]); // false

MyDto.accepts([]); // true

const myObject = MyDto.ensure([]);

console.log(myObject); // "{}"

@neos-project/framework-observable

The @neos-project/framework-observable package provides a very simple implementation of the Observable-pattern (see: https://github.com/tc39/proposal-observable).

Furthermore, based on Observables, it establishes two key primitives for use in the UI:

State

A State observable represents a value that changes over time. It can be asked about its current value and any subscriber will receive the current value immediately.

It also provides an update method, that allows to push the next value calculated from the current value.

Channel

A Channel observable represents a sort of message bus. A subscriber will receive all messages that have been published to the Channel since the subscription was established.

It provides a publish method to push messages onto the channel.

@neos-project/framework-react-hooks

The @neos-project/framework-react-hooks package contains a bunch of essential React hooks to allow binding state to React components.

Most notably it contains the useLatestValueFrom hook, that allows to bind Observables to components.

Code Generation

The predictable nature of the API primitives from above makes them easy to translate between different languages. Therefore, this PR contains a code generator that will automatically transform a set of API objects into TypeScript files ready for use in the Neos UI client code base.

The resulting files use @neos-project/framework-schema to build runtime-schemas from DTOs (and all inheriting concepts). Also, it uses @neos-project/framework-observable to create a Channel for each Notification.

A later idea would be to also provide a State for each Status.

There is a command controller, so that code generation can be triggered via:

./flow ui:generatepackages

This command will emit its result to the (newly established) @neos-project/neos-ui-api package.

To get an idea of the result, you can take a look at this commit: afa8381

Binding the generated code to the API: @neos-project/framework-api

The @neos-project/framework-api package takes care of all ceremony required, to make fetch-calls to the API.

The @neos-project/neos-ui-api package uses the methods from @neos-project/framework-api to create callable Queries and Commands.

@neos-project/framework-api also provides the method startPolling that creates the Inbox polling interval.

Re-Implementation of the Inspector

First of all, I have to admit: this is not a full re-implementation. My goal was to showcase at least one Query, at least one Command and at least one Notification. Therefore, I re-implemented the inspector only up to the point where these concepts could be shown.

This also means, that there's a lot of stuff that doesn't work (not because of the concept, but just because I didn't implement it). This includes:

• ClientEval: (too complicated to effectively showcase)
• SecondaryInspector (because it wasn't relevant for this)
• Changes from the inspector will not be recognized by the publishing menu (this would have meant an additional query, which I considered to be out of scope)
• etc.

So, what will work is focusing a node that has a TextEditor (or any other editor that doesn't require a secondary inspector), edit the respective property and hit "Apply". The property should be changed and if it is marked with reloadIfChanged it should also be updated in the ContentCanvas.

The new inspector implementation is supported by the following API objects:

GetInspectorQuery

Since the idea was to move responsibility from the client to the server, the server is now responsible for turning a node type configuration into the information necessary to render the Inspector. This is achieved by the GetInspectorQuery. It contains a reference to a node, which the GetInspectorQueryHandler then uses to generate the GetInspectorQueryResult - a full configuration for the Inspector.

Several factory classes assist this process, all of which can be reused for other queries, commands, notifications or statuses.

GetRenderedNodeQuery

This query exists to fetch an updated rendered content element in case of reloadIfChanged.

ChangeNodePropertiesCommand

This command is executed when the user clicks on "Apply". The ChangeNodePropertiesCommandHandler will publish a PropertiesWereUpdatedNotification once it is finished.

PropertiesWereUpdatedNotification

Immediately after the execution of every command, the Inbox of the current user is polled. After the ChangeNodePropertiesCommand, the Inbox will contain a PropertiesWereUpdatedNotification, that also tells the UI whether the change requires a reload of the affected content element.

One more thing: AlertNotification

To demonstrate truly asynchronous interaction, I added the AlertNotification. If you run the following flow command:

./flow ui:sendmessage --message "Hello World!"

A respective alert box will (eventually, takes up to 10 seconds) open in the UI.

[mficzel]

Can you explain why you opted for a custom api and not GraphQL. I personally like this approach here for the "simplicity" but the graphql question was asked in the weekly.

[mhsdesign]

And can you please also show you cool drawing of all the needed neos apis from ca. 1 year ago (i dont know if this still has relevance but for completeness)

[nezaniel]

Awesome <3

Do you already have an idea what the whole picture would look like if we inserted a ReactPHP server relaying messages from the CR?

[grebaldi]

@mficzel:

Can you explain why you opted for a custom api and not GraphQL

I will try to give an explanation for this, but will point out upfront, that I cannot offer much wisdom on this matter other than my own opinion. I'm sure this will remain a controversial point and I'd like to invite everyone who sees this differently to voice their ideas and concerns 🙂.

I've been approaching this implementation from a pure design perspective based on the design principles I've stated above. What I've ended up with is a message exchange protocol that simply doesn't require the use of GraphQL.

Of course, I would be remiss not to mention that every principle contained in this PR would easily translate into GraphQL terminology:

Primitive	GraphQL term
Query	Query
Command	Mutation
Notification/Status	Subscription

This means, that it would be perfectly possible to wrap a GraphQL facade around the concept presented here, but I would advise against that, because it would only add weight and offer no benefit (at least none that is apparent to me).

At the same time, I'd like to point out that GraphQL might as well be a very good fit for an entirely different design based on different design principles.

In fact, I believe that GraphQL would be ideal if our problem was to design what the DDD folks would call a "published language". That is a scenario in which the API would need to serve n clients, each with individual requirements that the API itself could not possibly predict. I'd be delighted to have something like that for the Content Repository 🙂. And GraphQL would be an absolutely marvellous format for it.

However, I do not believe that the Neos UI API fits into this description. The Neos UI is not just one application among others based on the Content Repository. It is the application to manage it (plus other concerns as well). The UI also shares its release cycle with the Neos core code base and there are plans to move it into the neos-development-collection monorepo. These are strong indicators, that the UI is not to be understood as an independent application.

I instead chose to look at the UI being in (what the DDD folks would call) a "customer-supplier" relationship with the Neos core. Here, the "supplier" (the Neos UI server code) fulfills the requirements of the "customer" (the Neos UI client code).

My design is therefore based on the idea of the backend-for-frontend pattern (cutely abbreviated to BFF 🙂), in which client and server are closely tied together. The idea is to put the server in the lead, having the responsibility to translate domain concepts from several sources into presentation models the client can easily understand. (Which serves the first design goal: Move responsibility from the client to the server)

Through code generation and automatic serialization/deserialization the message exchange protocol vanishes completely, so there's no need for a public schema. Everything is encoded in data structures and types native to the respective code bases. (Which serves the second design goal: End-to-end type safety)

The third design goal of extensibility is a bit more complicated to outline and admittedly, this PR does not show much of it. But in principle, any package could add new Query-, Command-, Notification- and Status-objects and use the same code generation mechanism to achieve client integration.

I would argue that for each of these design principles, GraphQL either has no answer or has nothing to add to what's already there.

Last but not least, I would to like to stress once more, that all of this is just my opinion. I'm not dogmatic about this and always open to discuss the merits of GraphQL or any alternative design 🙂.

---

@mhsdesign:

And can you please also show you cool drawing of all the needed neos apis from ca. 1 year ago

Yep, here's the big picture:

An here's the link to excalidraw: https://excalidraw.com/#room=9d6b9623b50fea60d3d9,GHI6rjZjT67_LoIWiTiz9g (I hope it works 😅)

This diagram is surely incomplete (and may be even outdated to some degree), but a lot of it is still valid. I need to point out though, that it might be confusing as to what it shows exactly. I've created it based on the status quo, but with the new design in mind. The green boxes roughly correspond to queries, the blue boxes roughly correspond to commands. This was meant as a basis to collect an inventory of requirements for API messages.

Thanks for the reminder, @mhsdesign 🙂

---

@nezaniel:

Do you already have an idea what the whole picture would look like if we inserted a ReactPHP server relaying messages from the CR?

Ooph... Didn't think of that 😅. I guess, it depends a bit on what kind of message exchange you have in mind there.

I can say this much: The API can be instantiated independently from the Flow HTTP stack. In fact, it doesn't have any hard dependencies to Flow at all. In theory, it could be easily integrated with a ReactPHP environment, to, say, publish notifications based on messages relayed from the CR. But the limits are with Flow's compatibility to such a scenario, because the infrastructure layer of the API is still bound to Flow.

Does this relate to your question?

[mficzel]

@grebaldi i 100% agree. I think the separation into Commands, Querys and Notifications is smart and fits our purpose well. Also the implementation as PHP DTOs with autogeneration of TS-Types.

The unified language between be and ui makes the actual format between pretty much an implementation detail we may even be able to change later on. GraphQL can do that offcourse but so does JSON/HTTP.

The drawback of GraphQL would be that there would be quite some tech between UI and Backend which makes debugging way harder than JSON/HTTP Messaging. On the other hand i see no real benefit for using GraphQL as this is an api specifically designed for the Neos UI will probably be of very limited use for other cases. Also i think that we will not be able to benefit from the overfetching prevention of graphQl. On the other hand using a very simple JSON/HTT messaging makes later optimizations like realtime socket communication much easier for future us.

GraphQL imho makes a very good general purpose API to the CR for other use cases and we should probably provide it as an option for frontend applications.

In total: We i think we should use Commands, Querys and Notifications as DTO in PHP and TS, as a dedicated API for the UI and not a generic one. Between those we should use the simplest toolset that is readily available.

[JamesAlias]

This is an amazing read ❤️ Thank you for sharing this proposal and even backing it up with actual implementations. I had lot's of "AHA" and "WHOA" moments reading it and skimming through the code 🤩

Since I learned about zod I wanted to think more about the validation of messages between systems and components (dev & runtime). Sadly I have not yet had time for it. I feel like it's kind of what you did with @neos-project/framework-schema, right? Do you think zod lacks features you definitely want to have, hence writing your own schema framework?

We recently had a discussion about NATS. Maybe this or a similar system can help to streamline the messaging/notification (you called it Inbox) system between backend(s) and frontend(s). Maybe it could even make cross user/device collaborative editing on Workspace-level possible. I'm not sure about caching and rollback, though. But that's just "fast thinking". I'd love a deeper, slower discussion about the topic.

That's my 2 cents 😄
Keep us updated with your thoughts on your proposal. I find it fascinating 🙂

[grebaldi]

Thanks a lot for your feedback @JamesAlias!

Since I learned about zod I wanted to think more about the validation of messages between systems and components (dev & runtime). Sadly I have not yet had time for it. I feel like it's kind of what you did with @neos-project/framework-schema, right?

That is correct. The problem that @neos-project/framework-schema tries to solve is the validation of arbitrary data against a defined schema at runtime whilst avoiding having to write that schema and the corresponding TypeScript type definition redundantly.

Do you think zod lacks features you definitely want to have, hence writing your own schema framework?

Quite the opposite :) It is true that this problem can potentially be solved using zod. In fact, there are a lot of libraries out there with a similar mission:

• runtypes
• io-ts
• yup
• Many, many more, I'm sure :)

The problem I see with these libraries (zod included) is their unclear feature boundary.

I'll try to illustrate that. Our message exchange looks like this:

Server               Message             Client
(PHP)   <--- [ JSON (stringified) ] --->  (JS)

Messages are encoded in JSON. On the PHP end, there's the API framework with it's DTO abstraction and invariant checks that will translate a message into PHP's type system. On the JS end, we have TypeScript, which provides static type-safety only, so we need a mechanism to verify that a received message corresponds to an expected shape in order to be sure that our static assumptions remain valid.

The messages our JS client can receive adhere to certain constraints that we know in beforehand:

1. A received message is an element of the set of all data structures that can be expressed in JSON
2. A received message also conforms with the DTO invariants as enforced by the API framework (This is by-design equivalent with the above point)
3. A received message has been encoded with PHP's json_encode function

This allows us to narrow the requirements, as we need a type verification library that...

1. ...is strictly limited to JSON-compatible data types (string, number, boolean, null, object/hashmap, array)
2. ...deals correctly and seamlessly with PHP json_encode-quirks (namely: empty PHP-arrays are ambigiuous to the encoder and may represent an empty JSON-object or an empty JSON-array)
3. ...produces helpful error messages (helpful to Neos-integrators that is)

Libraries like zod or the others mentioned above could be used to fulfill these requirements (with a little bit of compromise here and there). But besides our requirements, they'll do a lot more too, like:

• Validation of runtime constraints (like testing strings for regex-patterns, checking if a number is contained within a specific interval, etc.)
• Pattern matching
• Type branding
• Function contracts

Now, it might seem like a little bit of a stretch, but I would argue that the name of the adversary is entropy. If those extra features mentioned above are left open at the message boundary between client and server, this would potentially attract business logic to the wrong place (through refactoring, bugfixes, etc. - any code change over time). As a consequence, the UI may erode into strong cohesion with zod (or the respective alternative).

This could be solved by wrapping the third-party library inside an anti-corruption layer on our side, that would basically provide the same API as @neos-project/framework-schema does now. I think that would be fine as well, but intuitively I would also expect the maintenance burden to be greater, because we would have to keep track of a third-party library that covers a lot more concerns than just ours.

We recently had a discussion about NATS. Maybe this or a similar system can help to streamline the messaging/notification (you called it Inbox) system between backend(s) and frontend(s). Maybe it could even make cross user/device collaborative editing on Workspace-level possible. I'm not sure about caching and rollback, though. But that's just "fast thinking". I'd love a deeper, slower discussion about the topic.

Thanks for sharing NATS :)

Regarding the Inbox mechanism I have to admit this is the weakest part of the design. My goal was to create a mechanism for asynchronous messaging that does not require additional infrastructure for existing setups. The Inbox would achieve exactly that, but is not particularly elegant.

I have not put much thought into potential message-based features, but I believe that there's lots of potential. So, I'm definitely looking forward to discuss this topic in more detail as well :)