Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rewrite the backend using Valum #40

Closed
wants to merge 84 commits into from

Conversation

@arteymix
Copy link
Contributor

arteymix commented Nov 11, 2016

There's still some work to do, but I think it gives a good overview of what Valum can do.

I will probably integrate Template-GLib to remove all the inlined HTML someday.

Here's what should remain to do:

  • produce proper ETag for caching
  • finish deployment with Docker (or anything else we use)
  • finish backward compatibility
    • reimplement search.php
    • reimplement tooltip.php
  • test with sphinx
  • transition to nicer URIs? (e.g. /gio-2.0/GLib.File instead of ``gio-2.0/GLib.File/index.htm`)
  • install Valum on the CI (will require Trusty for Python 3.4 for Meson)
  • read and render templates asynchronously
  • query the database asynchronously (see http://jan.kneschke.de/projects/mysql/async-mysql-queries-with-c-api/ or use some thread-based approach)
  • implement a LRU cache to avoid reading from the filesystem
@arteymix arteymix force-pushed the arteymix:wip/backend-rewrite branch 5 times, most recently from 49b9137 to 75de385 Nov 11, 2016
@arteymix arteymix force-pushed the arteymix:wip/backend-rewrite branch from 75de385 to 0f7e4d8 Nov 14, 2016
@arteymix
Copy link
Contributor Author

arteymix commented Nov 16, 2016

Would it be reasonable to render search results server-side if the user does not have JavaScript? I think I could do something like that once I get everything working.

It's working without JavaScript quite nicely at this moment. I realized that the frontend is fetching some specific templates which are not served at this time.

Also, how comfortable would you be of serving this off Docker?

@lewisgoddard
Copy link
Member

lewisgoddard commented Nov 16, 2016

A server-side search results page would be a nice fallback.

Docker is fine for local testing, although we tend to use Vagrant at @elementary Deployment tends to occur on bare Digital Ocean installs.

@arteymix
Copy link
Contributor Author

arteymix commented Nov 16, 2016

Valum can be installed system-wide or used locally with Meson. I don't have Ubuntu packages yet, but I think people at elementary are good at this. If that works well, it will be ready for a final release.

I maintain a Docker image for deployment based on the latest Ubuntu: https://hub.docker.com/r/valum/valum/. It is really convenient, but I understand if you prefer to host it along existing apps.

We could have a Vagrantfile in this repository which replicate the hosting environment with Valum installed.

lewisgoddard and others added 4 commits Nov 16, 2016
@arteymix
Copy link
Contributor Author

arteymix commented Nov 18, 2016

It seems like the VM is using an old verison of libsoup, but Valum is has some fallback mode for that. In more recent versions, it will only warn for providing the interface property and use GIO SocketAddress API.

It's up to you all if you accept the branding for using Valum. I really don't mind removing it.

Explicitly use 'accept' to avoid serving static files with 'text/html'
content type.

Use a single 'serve_from_file' middleware for all static resources.

Use a single try-catch for both navigation and content templates.
@arteymix
Copy link
Contributor Author

arteymix commented Nov 18, 2016

It should be fully backward-compatible now.

There's probably a bug on the client side and lack of understanding this aspect prevents me from fixing it:

jquery.min.js:124 Uncaught DOMException: Failed to execute 'appendChild' on 'Node': Nodes of type '#document' may not be inserted inside nodes of type '#document-fragment'.(…)

I'll finish caching with ETag and look upstream with Meson to get the 0.36 working on Travis CI.

@arteymix
Copy link
Contributor Author

arteymix commented Nov 18, 2016

Feel free to test it and give feedback. I'm pretty much done here. For potential contributors, I wrote a lot of documentation: http://docs.valum-framework.org/en/latest/

@arteymix
Copy link
Contributor Author

arteymix commented Nov 18, 2016

There's still search and tooltip to reimplement :)

@arteymix arteymix force-pushed the arteymix:wip/backend-rewrite branch from 960e341 to 4b1b459 Nov 18, 2016
@arteymix arteymix force-pushed the arteymix:wip/backend-rewrite branch from 3ab096b to 5046c11 Nov 18, 2016
@arteymix arteymix force-pushed the arteymix:wip/backend-rewrite branch from 5046c11 to b0bf60d Nov 18, 2016
@arteymix arteymix force-pushed the arteymix:wip/backend-rewrite branch from 7137b1e to fbcec79 Feb 24, 2017
@arteymix
Copy link
Contributor Author

arteymix commented Feb 24, 2017

I added a hidden input for the search page to include the package, which is less hackish than checking the current URL.

It's pretty much all functional with fixed @ in symbols.

We have to be careful when merging because the main template is embedded in the code and I think there's been some good UI work already.

@benwaffle
Copy link
Member

benwaffle commented Jun 18, 2017

Can we add glrucache in-tree? Or maybe use meson for the vala code and just have valum, compose, and glrucache be subprojects?

@benwaffle
Copy link
Member

benwaffle commented Jun 18, 2017

Also I'd like a dockerfile for development so I don't have to have sphinx installed. Vagrant uses virtualbox which is kinda heavy

@arteymix
Copy link
Contributor Author

arteymix commented Jun 18, 2017

We can add a Dockerfile. I think that if we stick on the same distribution, we can have a common provision script and avoid duplicating all the initialization.

I did not introduce Meson to keep this as a drop-in replacement. If we do this, then it will make a lot of sense to use the compatible dependencies via subprojects.

EDIT: I submitted this patch https://github.com/chergert/glrucache/pull/6 so we can use it properly and I have a local version that current work using Meson

@benwaffle
Copy link
Member

benwaffle commented Oct 25, 2017

Lets use meson subprojects for valum/vsgi/glrucache like this: https://github.com/benwaffle/pulse-flow/blob/master/subprojects/libgtkflow.wrap

@desiderantes
Copy link
Contributor

desiderantes commented Jun 24, 2018

Sad to see this incomplete:( Is it still viable?

@ZanderBrown
Copy link
Contributor

ZanderBrown commented Jun 24, 2018

Hopefully this can be finished, so much could be done with a JSON/XML API built on top of this

@arteymix
Copy link
Contributor Author

arteymix commented Jun 24, 2018

@ZanderBrown I'm really busy right now with my research project and I truly hope I could put some time on this side-project by the end of the summer.

If anyone want to take the lead, I'll be glad to coordinate the efforts.

@ZanderBrown
Copy link
Contributor

ZanderBrown commented Jun 25, 2018

If we have a clear TODO i'm willing to try & find time for a couple items

@arteymix
Copy link
Contributor Author

arteymix commented Jun 25, 2018

I'll upload a fully working Vagrant setup tomorrow, fix the merge conflicts and assess the remaining work to be done into a proper TODO list.

I think it's okay to use the coming VSGI 1.0 and Valum 0.4 which are already near stabilization.

@benwaffle
Copy link
Member

benwaffle commented Jun 25, 2018

https://github.com/eustasy/vala-docs.org - this generates an XML file containing all of the info needed. Would be good if the valum backend used that

@btkostner
Copy link
Member

btkostner commented Jul 12, 2018

The more and more I think about this, I think the best route would be to have a Valum backend that uses the XML file generator to create an API only backend server.

Then having a frontend project that consumes the API just like every other client (builder, docs, etc.)

This keeps the backend all in vala. Easy to edit and maintain for vala devs. Also gives us more freedom with the frontend to offer some cool things like an pwa, and offline support. Maybe even an easier time including guides and less API-y like documentation.

EDIT:
Also, if we build the backend with Docker, we could potentially have mutliple backend servers up for different vala versions (if we really wanted that.). 0-36.api.valadoc.org

@arteymix
Copy link
Contributor Author

arteymix commented Jul 12, 2018

For Docker, I intended to setup a docker-compose environment to run the sphinx search engine and the HTTP backend separately. I don't know if that is consistent with your future hosting goals, but that could be a nice-to-have setup for scaling and targeting other hosting providers.

The Vagrant setup works for testing purposes, but I'm afraid I cannot put more time and efforts on this side project for now.

@btkostner
Copy link
Member

btkostner commented Jul 12, 2018

Eventually down the line I would love to have everything just running in docker containers on kubernetes, but it's going to be a while before that happens. I was thinking as far as deployment goes, having CI create a docker image that contains all of the compiled vala code and docs as that time. That way if you need to go back in time and look at previous docs for some reason, it's just changing the docker image version. This also makes it a lot easier to deploy and pull down locally if you need.

@tintou
Copy link
Member

tintou commented Oct 28, 2019

I'll close it as there is no activity since more than a year, feel free to reopen when ready

@tintou tintou closed this Oct 28, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

9 participants
You can’t perform that action at this time.