[WIP] Foxx Of The Future

[pluma]

This PR tracks the development of the Foxx rewrite for ArangoDB 3.0:

• applicationContext -> module.context
• manifest.json#controllers/exports -> manifest.json#main
• manifest.json#files#type for hardcoded mime types (see Manually set mime types #1702)
• replace arangodb.guessContentType with mime-types everywhere
• remove manifest.json#setup/teardown
• add manifest.json#provides (3.x)
• handle manifest.json#provides for manifest.json#dependencies in Aardvark & Foxx CLI (3.x)
• implement Routers
  • implement createRouter
  • implement Router#get/post/put/patch/delete
  • implement Router#all
  • implement Router#use
  • implement FoxxContext#use
  • implement Tree
  • implement Tree#resolve
  • implement Tree#dispatch
  • implement Treer#buildSwaggerPaths
  • implement request handler in FoxxService
  • implement FoxxContext#reverse
  • implement synthetic request/response objects
  • implement request body handling
  • implement response body handling
• implement body parsers/serializers
• implement legacy compatibility layer for ArangoDB 2.x Foxx apps
• move Foxx.Repository into legacy
• move Foxx.Model into legacy
• move Foxx.Controller into legacy
• convert Foxx generator
• move OAuth2 into Foxx
• move auth-simple into Foxx
• move sessions-local into Foxx
• move users-local into Foxx
• migrate system services
• refactor implementations (3.x)

Wishlist:

• handle multipart/form-data
• handle recursive multipart request bodies
• produce more granular (415/422) errors for illegal request bodies

[pekeler]

Any chance we can have the old Foxx framework coexist with new framework, perhaps by using a different namespace? Not forever, just for a release. Using ArangoDB in production, I dislike the idea of deploying a new major version of the DBMS and also of my Foxx app at the same time. I'd rather first upgrade to ArangoDB 3, and only when it's been running stable for a few days upgrade my Foxx app.

[pluma]

@pekeler Foxx.Repository and Foxx.Model can be implemented on top of the new API. It would probably also be possible to provide a wrapper for the new Router that behaves similarly to the current Foxx.Controller.

We're looking into codemods using Facebook's jscodeshift as an option to make the conversion of "classic" Foxx apps to ArangoDB 3.0 services easier but can't promise anything at the present time.

We will try to make the migration as easy as possible.

[pekeler]

My concern is more about the duration of downtime and risk mitigation when upgrading production, which is why I'd prefer to upgrade the DB and my Foxx app in two separate steps.

Any assistance in converting my code is appreciated but not as much of a concern to me.

[pluma]

@pekeler You mean having ArangoDB 2.8 Foxx apps work with no modifications at all on ArangoDB 3.0?

I think we might be able to implement a compatibility layer and switch based on the "engines" field in the manifest (i.e. if arangodb is present and set to a version lower than 3, try to wrap it in the compatibility layer) but I'll need to look deeper into this.

[pekeler]

You mean having ArangoDB 2.8 Foxx apps work with no modifications at all on ArangoDB 3.0?

Yes.

[pluma]

@pekeler Okay, the plan is that we'll make it possible to run ArangoDB 2.8 services in ArangoDB 3.0.

In order to do this without hauling around a lot of baggage we'll backport a few changes to 2.8 (which itself aims to be backwards compatible with 2.7, 2.6 and 2.5 within reason).

This means the upgrade path looks as follows:

1. Upgrade database to 2.8
2. Upgrade Foxx services to use future-proof subset (e.g. not rely on magical comments) and set engines field in manifest to "arangodb": "^2.8.0"
3. Upgrade database to 3.0
4. Upgrade Foxx services to new Foxx API

Because 2.8 will support the necessary changes to make legacy services work in 3.0 this allows using 2.8 as a stop gap and upgrading to 3.0 by letting the 2.8 services run in compatibility mode before upgrading them to 3.0 proper.

[pekeler]

Great 👍 Thank you!

[pluma]

✨ 🎉 ✨

[baslr]

@pluma to continue on #1612 (comment)
I asked for res.send because Im interested in EventSource https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events

foxx example could be:

var eventsource = null;

// init event source
controller.get('/eventsource', {end:false}, (req, res) => {
  eventsource = res;
  res.setHeader('content-type', 'text/event-stream');
});

controller.get('/streamresult', (req, res) => {
  res.end();
  var it = db.query('for user in users return user');
  var p = undefined;
  while(p = it.next() ) {
    eventsource.wirte('event: user');
    eventsource.write(JSON.stringify(p);
  } // while
});

We only need to control if the response is auto ended and to set headers.

[pluma]

@baslr the API changes are currently "cosmetic", i.e. because the underlying internals of ArangoDB's request cycle haven't changed it's still not possible to send responses incrementally. We currently do not have support for EventSource/SSE.

Also, the code example you're proposing would not work because there's no guarantee the requests to eventsource and streamresult would be handled in the same thread/context (so eventsource might still be undefined) and because responses are automatically ended when the Foxx request handler terminates (i.e. the res from the first handler would be terminated by the time the second handler tries to write to it) -- not to mention that the behaviour would be highly unreliable when multiple clients make requests to the first endpoint (writing over each other if they do end up in the same context).

We're looking into several possible implementations of realtime streams at the moment but I do not think they will be available in the 3.0 release. If you want to discuss streams further (and make a case for eventsource vs websockets, for example), feel free to follow issue #602.