API V2

[NicolasCARPi]

When I created the API v1 back in the day, it was very much hectic and grew organically. It is now time for a V2 with a better design to stick to modern/best practices when it comes to APIs.

The idea would be to fully use all HTTP verbs properly, and to have a clean interface that actually makes sense. To be honest the V1 is not too shabby, but there is room for improvements and there are many actions that are unavailable.

The end goal would be to have the frontend interface actually use the API in XHR requests directly. This would be nice.

So instead of directly starting coding as I often do. This time I'll try to think before I do. And share it publicly because your comments will be welcome.

Design for API V2

OUTDATED: see updated design here: #3642 (comment)

Creating stuff

request: POST /experiments
response: 201 Created with the URL to new entity in Location header

Reading stuff

This is similar to v1.

request: GET /items/12
response (you'd get the full entity as json)

{
    "title": "entity title",
    "body": "blah..."
}

Without an id you get an array of items (like v1).

Updating stuff

PATCH /experiments/12

{
    "title": "new title"
}

And the response would be the full entity. Using PUT you'll need to provide a full entity.

Delete stuff

request: DELETE /templates/12
response: 204

Errors

{
    "code": 12434,
    "message": "Something bad",
    "description": "rasiteriuse truiste ",
    "errors": [
        {
            "code": 444,
            "field": "date",
            "message": "Incorrect date format",
        }
    ],
}

Doing an action

Like timestamping an entity

POST /experiments/123/timestamps
201 Created

Endpoints

Similar to v1 with experiments, items, experiments_templates, items_types, but with new ones like users, teams, etc....

Actions would be CRUD for all and some would get more like comments and lock toggle or archive actions.

Conclusion

There is no "one true way" to design an api, and every approach has shortcomings. But I think this looks pretty neat, what do you think?

[HenningTimm]

Very cool to see work on a new API version. I am looking forward to this 🙂 Here are some things that crossed my mind when reading this. I am not an experienced API programmer (and an even less experienced API developer 😉) though, so please feel free to educate me on obvious errors and false assumptions I make.

HTTP verbs

I am a big fan of using different verbs for creating and updating entities. I think this nicely prevents accidentally overwriting existing experiments etc. If I understood correctly, creating and updating an experiment would work as follows:

1. POST /experiments This gives me a new id. Can I already supply content for the experiment or should this be only the creation of an empty entity?
2. PUT /experiments/id Using this I hand over a complete JSON record of the entity I want to update. If the record is incomplete or the id is invalid I get a respective HTTP error (e.g. 403 for wrong permissions, 404 for an unavailable id, ...).
3. PATCH /experiments/id Using this I can update single fields of the entity, depending on the content of the JSON payload I provide. (So, if I provide a full entity as payload, this would do the same as PUT, but when using PUT I get an error if I miss/ forget an entry.)

Did I get that right?

Endpoints

Toggles vs. explicitly setting states

What would you propose for changing boolean states of entities? E.g. to lock and unlock an experiment, would that be POST /experiments/123/lock to lock and POST /experiments/123/lock again to unlock? Or should that be two different endpoints POST /experiments/123/lock and POST /experiments/123/unlock (which seems complicated) or a single endpoint with a parameter like { "lock": True } or { "lock": False }.

Update: After reading up on this, toggles seem to be inherently unRESTful, since they are not idempotent. So probably using just one lock endpoint with a passed JSON parameter would be the way to go? In the linked SO question, there is a suggestion to use two different endpoints when the different actions require different permissions (on the access token level).

[NicolasCARPi]

Can I already supply content for the experiment or should this be only the creation of an empty entity?

I think it will be in two steps. But providing a template id should be possible.

If the record is incomplete or the id is invalid I get a respective HTTP error

Yes you'll get proper http errors (like in v1) and if you PUT something incomplete I guess the non-present fields gets nulled? Sounds dangerous. So yeah maybe erroring out if fields are missing could be best.

So, if I provide a full entity as payload, this would do the same as PUT, but when using PUT I get an error if I miss/ forget an entry.

Indeed.

For actions such as lock, yes it is unrestful but that's how life is ;) I also read a bunch of SO posts about it. Maybe something like if you don't provide a lock: bool, it toggles, and if you provide it becomes idempotent?

[anargam]

For actions such as lock, yes it is unrestful but that's how life is ;) I also read a bunch of SO posts about it. Maybe something like if you don't provide a lock: bool, it toggles, and if you provide it becomes idempotent?

Wouldn't it be easier just to design it to be idempotent to start with? Ie a successful request to lock should result in the entity being put in a locked state, independent of the previous state? Simplifies eg resend semantics, handling at least almost-concurrent users (there are still dragons here, I suspect)... Or perhaps I'm misinterpreting the purpose of the action (is this a setlockstate endpoint that requires a boolean describing the state? A toggle sounds like it performs lockstate := ! \old(lockstate)).

[NicolasCARPi]

the current web behavior and php code is a toggle. But I agree with you doing something idempotent is better. So it should be /experiments/12/lock and /experiments/12/unlock as a POST.

[MarcelBolten]

API V2 seems to be a great chance to also split the uploads table in two parts, one for items and one for experiments as outlined in #3393 if that is still desirable.

The current API requires all upload ids in one table in order to GET /uploads/:id. Not sure what would be the best way to request an upload and pass along the entity type. Maybe GET /experiments/uploads/:id and GET /items/uploads/:id for a single file and GET /experiments/:id/uploads to get all attached uploads.

That lets me think if it should be possible to request only a filed like body or next_step via GET /experiments/:id/body compared to request the full entry via GET /experiments/:id.

[NicolasCARPi]

split the uploads table in two parts

Agreed.

Maybe GET /experiments/uploads/:id

Yes!

if it should be possible to request only a field like body

No. While it is possible now in the code, it adds overhead and the full entity is read anyway (and then only the column is filtered in php and sent back). It's best to send all and let the consuming code pick up only what they need.

[NicolasCARPi]

So the APIv2 rework is going well, currently here is how it looks:

1. The APIv2 is directly accessible from the browser, too (if you're logged in), so you can GET /experiments/42 and see the JSON directly.
2. APIv1 will still be available and functional in the next version, but deprecated.
3. The web interface uses API calls directly now.
4. The current diff for the apiv2 branch related to hypernext branch is 292 files changed, 5825 insertions(+), 8034 deletions(-). Yes, many lines were deleted which is delightful. A lot of the code has been revamped to fit the new API design. I'm pretty excited to bring these changes to the codebase, even if they won't directly be visible to end-users, it's always a good thing for a software project to address technical debt and remove accumulated cruft.
5. The documentation is not yet available but will be at release time.

Creating stuff:

POST which replies a 201 (Created) and the location of the created entity in the location header of the response. You can send additional parameters like a template_id for experiments when creating, or tags.

Some creative actions like Duplicate also goes through POST with an "action: duplicate" parameter.

Reading stuff:

GET which replies in JSON. You can use the format query parameter to get a pdf, eln, csv, or zip output instead! Or use the q param to search for something.

Updating stuff

PATCH which replies with the updated entry (complete). You can patch one or several columns at the same time.

The default action of patch is update, but some other actions are possible like "lock" or "timestamp". For that you send a patch with "action: timestamp" in the body for instance.

Deleting stuff

DELETE which replies 204.

Endpoints:

Might be easier to show current code:

elabftw/src/controllers/Apiv2Controller.php

Lines 189 to 252 in a6cf80d

	private function getModel(): RestInterface
	{
	switch ($this->endpoint) {
	case 'apikeys':
	return new ApiKeys($this->Users, $this->id);
	case 'config':
	return Config::getConfig();
	case 'experiments':
	case 'items':
	case 'experiments_templates':
	case 'items_types':
	return (new EntityFactory($this->Users, $this->endpoint, $this->id))->getEntity();
	// for a single event, the id is the id of the event
	case 'event':
	return new Scheduler(new Items($this->Users), $this->id);
	// otherwise it's the id of the item
	case 'events':
	$defaultStart = '2018-12-23T00:00:00+01:00';
	$defaultEnd = '2119-12-23T00:00:00+01:00';
	return new Scheduler(
	new Items($this->Users, $this->id),
	null,
	(string) $this->Request->query->get('start', $defaultStart),
	(string) $this->Request->query->get('end', $defaultEnd),
	);
	case 'team_tags':
	return new TeamTags($this->Users, $this->id);
	case 'teams':
	return new Teams($this->Users, $this->id);
	case 'todolist':
	return new Todolist($this->Users->userData['userid'], $this->id);
	case 'users':
	return new Users($this->id, null, $this->Users);
	default:
	throw new ImproperActionException('Invalid endpoint.');
	}
	}
	private function getSubModel(string $submodel): RestInterface
	{
	if ($this->Model instanceof AbstractEntity) {
	return match ($submodel) {
	'comments' => new Comments($this->Model, $this->subId),
	'links' => new Links($this->Model, $this->subId),
	'steps' => new Steps($this->Model, $this->subId),
	'tags' => new Tags($this->Model, $this->subId),
	'uploads' => new Uploads($this->Model, $this->subId),
	default => throw new ImproperActionException('Incorrect submodel.'),
	};
	}
	if ($this->Model instanceof Teams) {
	return match ($submodel) {
	'status' => new Status($this->Model, $this->subId),
	'teamgroups' => new TeamGroups($this->Users, $this->subId),
	default => throw new ImproperActionException('Incorrect submodel.'),
	};
	}
	if ($this->Model instanceof Users) {
	return match ($submodel) {
	'notifications' => new Notifications($this->Users, $this->subId),
	default => throw new ImproperActionException('Incorrect submodel.'),
	};
	}
	throw new ImproperActionException('Incorrect endpoint.');

Basically you have more endpoints than in v1, and you have subendpoints/submodels. So to create a comment you can POST experiments/42/comments, and update it with PATCH experiments/42/comments/23

It's not finished but the heavy work has been done already :)

[MarcelBolten]

@NicolasCARPi
In the current version 4.3.10 the quick search on the experiment/database pages can be used with the fields from the search page. I noticed that this feature is gone for now in 4.4.0. I assume this is partially due to the requirement to search title, body, elabid and date but the default of the search page is only title and body. If this is the only reason why the quick search behaviour was changed I think we could adjust the default for the quick search.

Of course, this will affect the API too.
I think it will be a bit cumbersome to use either the 'q' and/or 'extended' parameter to be able to fine tune a search result. Using only 'q' seems more user friendly to me.

[NicolasCARPi]

Yes, using only q is best.

[jat255]

you'll probably be able to use "advanced" search

Great to hear! In the API docs for the GET request, it would probably be helpful to explicitly call out this option, perhaps with a link to the search syntax definition, and perhaps a couple of examples, so they all appear in the Swagger UI.

[NicolasCARPi]

@jat255 Just added the extended query parameter, works with v1 and v2. See: https://doc.elabftw.net/api/v2/#/Experiments/read-experiments

[MarcelBolten]

Yes, using only q is best.

I think there should only be q and no extended because q could be parsed like extended gets currently so there would be no need for extended.

[NicolasCARPi]

Documentation is taking shape slowly but surely. Swagger/Openapi is pretty great TBH.

[NicolasCARPi]

Nearly complete documentation can be found here: https://doc.elabftw.net/api/v2/