Rewrite the backend using Valum

[arteymix]

There's still some work to do, but I think it gives a good overview of what Valum can do.

I will probably integrate Template-GLib to remove all the inlined HTML someday.

Here's what should remain to do:

• produce proper ETag for caching
• finish deployment with Docker (or anything else we use)
• finish backward compatibility
  • reimplement search.php
  • reimplement tooltip.php
• test with sphinx
• transition to nicer URIs? (e.g. /gio-2.0/GLib.File instead of ``gio-2.0/GLib.File/index.htm`)
• install Valum on the CI (will require Trusty for Python 3.4 for Meson)
• read and render templates asynchronously
• query the database asynchronously (see http://jan.kneschke.de/projects/mysql/async-mysql-queries-with-c-api/ or use some thread-based approach)
• implement a LRU cache to avoid reading from the filesystem

[arteymix]

Would it be reasonable to render search results server-side if the user does not have JavaScript? I think I could do something like that once I get everything working.

It's working without JavaScript quite nicely at this moment. I realized that the frontend is fetching some specific templates which are not served at this time.

Also, how comfortable would you be of serving this off Docker?

[lewisgoddard]

A server-side search results page would be a nice fallback.

Docker is fine for local testing, although we tend to use Vagrant at @elementary Deployment tends to occur on bare Digital Ocean installs.

[arteymix]

Valum can be installed system-wide or used locally with Meson. I don't have Ubuntu packages yet, but I think people at elementary are good at this. If that works well, it will be ready for a final release.

I maintain a Docker image for deployment based on the latest Ubuntu: https://hub.docker.com/r/valum/valum/. It is really convenient, but I understand if you prefer to host it along existing apps.

We could have a Vagrantfile in this repository which replicate the hosting environment with Valum installed.

[arteymix]

It seems like the VM is using an old verison of libsoup, but Valum is has some fallback mode for that. In more recent versions, it will only warn for providing the interface property and use GIO SocketAddress API.

It's up to you all if you accept the branding for using Valum. I really don't mind removing it.

[arteymix]

It should be fully backward-compatible now.

There's probably a bug on the client side and lack of understanding this aspect prevents me from fixing it:

jquery.min.js:124 Uncaught DOMException: Failed to execute 'appendChild' on 'Node': Nodes of type '#document' may not be inserted inside nodes of type '#document-fragment'.(…)

I'll finish caching with ETag and look upstream with Meson to get the 0.36 working on Travis CI.

[arteymix]

Feel free to test it and give feedback. I'm pretty much done here. For potential contributors, I wrote a lot of documentation: http://docs.valum-framework.org/en/latest/

[arteymix]

There's still search and tooltip to reimplement :)

[arteymix]

I added a hidden input for the search page to include the package, which is less hackish than checking the current URL.

It's pretty much all functional with fixed @ in symbols.

We have to be careful when merging because the main template is embedded in the code and I think there's been some good UI work already.

[benwaffle]

Can we add glrucache in-tree? Or maybe use meson for the vala code and just have valum, compose, and glrucache be subprojects?

[benwaffle]

Also I'd like a dockerfile for development so I don't have to have sphinx installed. Vagrant uses virtualbox which is kinda heavy

[arteymix]

We can add a Dockerfile. I think that if we stick on the same distribution, we can have a common provision script and avoid duplicating all the initialization.

I did not introduce Meson to keep this as a drop-in replacement. If we do this, then it will make a lot of sense to use the compatible dependencies via subprojects.

EDIT: I submitted this patch https://github.com/chergert/glrucache/pull/6 so we can use it properly and I have a local version that current work using Meson

[benwaffle]

Lets use meson subprojects for valum/vsgi/glrucache like this: https://github.com/benwaffle/pulse-flow/blob/master/subprojects/libgtkflow.wrap

[desiderantes]

Sad to see this incomplete:( Is it still viable?

[ZanderBrown]

Hopefully this can be finished, so much could be done with a JSON/XML API built on top of this

[arteymix]

@ZanderBrown I'm really busy right now with my research project and I truly hope I could put some time on this side-project by the end of the summer.

If anyone want to take the lead, I'll be glad to coordinate the efforts.

[ZanderBrown]

If we have a clear TODO i'm willing to try & find time for a couple items

[arteymix]

I'll upload a fully working Vagrant setup tomorrow, fix the merge conflicts and assess the remaining work to be done into a proper TODO list.

I think it's okay to use the coming VSGI 1.0 and Valum 0.4 which are already near stabilization.

[benwaffle]

https://github.com/eustasy/vala-docs.org - this generates an XML file containing all of the info needed. Would be good if the valum backend used that

[btkostner]

The more and more I think about this, I think the best route would be to have a Valum backend that uses the XML file generator to create an API only backend server.

Then having a frontend project that consumes the API just like every other client (builder, docs, etc.)

This keeps the backend all in vala. Easy to edit and maintain for vala devs. Also gives us more freedom with the frontend to offer some cool things like an pwa, and offline support. Maybe even an easier time including guides and less API-y like documentation.

EDIT:
Also, if we build the backend with Docker, we could potentially have mutliple backend servers up for different vala versions (if we really wanted that.). 0-36.api.valadoc.org

[arteymix]

For Docker, I intended to setup a docker-compose environment to run the sphinx search engine and the HTTP backend separately. I don't know if that is consistent with your future hosting goals, but that could be a nice-to-have setup for scaling and targeting other hosting providers.

The Vagrant setup works for testing purposes, but I'm afraid I cannot put more time and efforts on this side project for now.

[btkostner]

Eventually down the line I would love to have everything just running in docker containers on kubernetes, but it's going to be a while before that happens. I was thinking as far as deployment goes, having CI create a docker image that contains all of the compiled vala code and docs as that time. That way if you need to go back in time and look at previous docs for some reason, it's just changing the docker image version. This also makes it a lot easier to deploy and pull down locally if you need.

[tintou]

I'll close it as there is no activity since more than a year, feel free to reopen when ready