Clarifications on how config.js is used

[olivierdalang]

Hi !

As discussed in wq/wq-django-template#20, I'm interested in contributing a docker-compose setup for WQ.io, so it could be used easily cross-platform and would be easier to deploy.

There is however one thing that I don't understand and which looks a bit contradictory regarding config.js,

It seems config.js is both :

• generated by a REST service (e.g. wq.db) - which tends to indicate that config.js is served dynamically from wq.db and loaded on runtime by the javascript app.
• manually generated with the dump_config management command (that is called from deploy.sh) - which tends to indicate that this is needed at build time for the javascript app.

I don't really see how these can both be true ?

This has an influence on how things can be containerized (ideally there would a nodejs image that can build completely independently from the django backend, but if the nodejs image requires config.js, it would get a little more complicated.)

Could you clarify how this works ?

[sheppard]

Both descriptions are essentially correct, though I agree the documentation should be improved. When the app is initialized, the wq config object is loaded from a static file, data/config.js. This file can be generated by wq.db's dump_config management command, or (as of 1.3.0b1), automatically whenever ./manage.py runserver detects a file change.

Once the user logs in, a second copy of the configuration is loaded dynamically. This version includes the base configuration plus any user-specific permissions. Thus, it is usually loaded via the wq.db's auth API, though is also available at /config.json. @wq/app surfaces the static configuration as app.config, and the dynamic one as app.wq_config. In wq 2.0, I plan to reduce the second copy to include only the permissions info, to reduce confusion about which copy is "the" configuration.

Early on, wq.db actually included an "AMDRenderer", making it possible to load the dynamic configuration directly as a JavaScript file (/config.js). This was removed in wq/wq.db#34 in to encourage the use of dump_config in production environments. The static configuration is used to set up the client-side URL router, so it really needs to be available immediately when the application loads. That said, it is still possible to defer initialization until the dynamic config is fetched, slightly degrading startup time:

  import wq from './wq.js';
  import somePlugin from './somePlugin.js';
- import config from './data/config.js';

  wq.use(somePlugin);

- wq.init(config);
+ async function init() {
+     const response = await fetch("/config.json"),
+         config = await response.json();
+     wq.init(config);
+ }
+
+ init();

As far as containerization, the first thing to note is that there are now two alternative templates, one of which does not require a JavaScript build system at all (see wq.app v1.3.0a1). My ultimate goal is to phase out deploy.sh in favor of just ./manage.py collectstatic or ./manage.py runserver for most simple use cases. Relying entirely on Python/Django for scripting will help solve the cross-platform deployment issue that prompted this discussion.

That said, power users will still appreciate the ability to customize a build with npm. For those users, you would indeed need to come up with a way to retrieve data/config.js from the django backend. Currently, @wq/cra-template won't even build until that file is in place. I would be open to discussing alternative solutions to make this work better, whether that means returning to a dynamic fetch or something else.

FWIW, I personally have found the pre-built static/app/js/wq.js to be more convenient than react-scripts + @wq/cra-template anyway, and have been using the --without-npm option for most of my projects. When I do need to add custom components to a project (which is almost always the case), I've been using @wq/rollup-plugin to create project-specific Python wheels containing reusable Django apps with pre-built wq JavaScript plugins in their static/app/js/ folders. Once installed and added to INSTALLED_APPS, these plugins can be deployed together with wq.js via ./manage.py collectstatic.

Not sure if that exact approach would work for you, but a Node+Python Docker container for building such a wheel could be pretty useful for my workflow.

[olivierdalang]

Thanks for the clarifications !

I like the idea of loading it dynamically only, as it reduces the complexity of the stack (static is really static :-) ). Wouldn't it be possible to push config.json with http2 to avoid initial round trip, yet still only have the dynamically generated config.json ? Or alternatively, inline it in index.html (obviously would only work if index.html is served dynamically) ?

My ultimate goal is to phase out deploy.sh in favor of just ./manage.py collectstatic

Yes that would be a great improvement ! And not necessarily hard to achieve, maybe you just just put the whole front-end code under another django app (with just static files), so it gets automatically found, is easy to override if needed, and you get all django advantages with it (served with test server, cdn integration, etc.)

In any case, I must say I'm a bit struggling with the current setup. Unfortunately I didn't manage to reuse the apache config as is (apache and mod_wsgi is quite a pain in a docker context compared to a better containerized solution with something like uwsgi + Caddy, which also fully takes care of https). But the apache config isn't easy to adapt (with semi-magic stuff like Options FollowSymLinks Multiviews and hard coded rewrites for static files). Your goal above would definitely help.

Do you have an est. timeframe for that ?

[sheppard]

Thanks for the feedback. I'd like to try and make config.js fully dynamic for the final 1.3 release, and also deprecate deploy.sh. A lot of the work is done already:

• wq.app is already a staticfiles-ready Django app as of wq.app 1.3.0a1.
• Most wq command-line tools are already deprecated as of wq.app 1.3.0b1.
• data/config.js is automatically refreshed by ./manage runserver as of wq.db 1.3.0b1, though it is still served as a "static" file.

Here's what still needs to be done:

1. Adapt the few remaining command-line tools into a custom ./manage.py collectstatic backend.
2. Restore config.js as a live API endpoint and deprecate data/config.js generation
   • The Django-only template could just have something like import config from '/config.js';
   • The CRA app uses Webpack, so it will probably need to do something like fetch('/config.json').then(init)

The main challenge with removing the static data/config.js file is that we'll still need to communicate where the WSGI service is (particularly if it's not at the root of the domain). One option would be to use relative paths to traverse out of the /static/app/js folder: import config from '../../../config.js'. Another option might be to directly modify index.js (or a separate glue file) during collectstatic, to replace references to /config.js with the configured site root. That will make it not really "static", but perhaps it is a reasonable compromise.

As for the Apache config, I think FollowSymLinks and Multiviews are just there since they are common defaults - there's nothing specific to wq that requires them. The hard-coded rewrites are there so everything can be under the same root URL, while still letting Apache optimize the delivery of static files. You wouldn't need to do that if you are going to use different root URLs, or aren't concerned with the WSGI overhead for static files.

[sheppard]

The latest wq.db release supports a /config.js endpoint that can be used as a fully live configuration. The template still uses the generated data/config.js file by default, but it's easy enough to change the template to reference the live version.