Rewrite the backend using Valum #40

Open
wants to merge 84 commits into
from

Conversation

Projects
None yet
8 participants
@arteymix
Contributor

arteymix commented Nov 11, 2016

There's still some work to do, but I think it gives a good overview of what Valum can do.

I will probably integrate Template-GLib to remove all the inlined HTML someday.

Here's what should remain to do:

  • produce proper ETag for caching
  • finish deployment with Docker (or anything else we use)
  • finish backward compatibility
    • reimplement search.php
    • reimplement tooltip.php
  • test with sphinx
  • transition to nicer URIs? (e.g. /gio-2.0/GLib.File instead of ``gio-2.0/GLib.File/index.htm`)
  • install Valum on the CI (will require Trusty for Python 3.4 for Meson)
  • read and render templates asynchronously
  • query the database asynchronously (see http://jan.kneschke.de/projects/mysql/async-mysql-queries-with-c-api/ or use some thread-based approach)
  • implement a LRU cache to avoid reading from the filesystem
@arteymix

This comment has been minimized.

Show comment
Hide comment
@arteymix

arteymix Nov 16, 2016

Contributor

Would it be reasonable to render search results server-side if the user does not have JavaScript? I think I could do something like that once I get everything working.

It's working without JavaScript quite nicely at this moment. I realized that the frontend is fetching some specific templates which are not served at this time.

Also, how comfortable would you be of serving this off Docker?

Contributor

arteymix commented Nov 16, 2016

Would it be reasonable to render search results server-side if the user does not have JavaScript? I think I could do something like that once I get everything working.

It's working without JavaScript quite nicely at this moment. I realized that the frontend is fetching some specific templates which are not served at this time.

Also, how comfortable would you be of serving this off Docker?

@lewisgoddard

This comment has been minimized.

Show comment
Hide comment
@lewisgoddard

lewisgoddard Nov 16, 2016

Member

A server-side search results page would be a nice fallback.

Docker is fine for local testing, although we tend to use Vagrant at @elementary Deployment tends to occur on bare Digital Ocean installs.

Member

lewisgoddard commented Nov 16, 2016

A server-side search results page would be a nice fallback.

Docker is fine for local testing, although we tend to use Vagrant at @elementary Deployment tends to occur on bare Digital Ocean installs.

@arteymix

This comment has been minimized.

Show comment
Hide comment
@arteymix

arteymix Nov 16, 2016

Contributor

Valum can be installed system-wide or used locally with Meson. I don't have Ubuntu packages yet, but I think people at elementary are good at this. If that works well, it will be ready for a final release.

I maintain a Docker image for deployment based on the latest Ubuntu: https://hub.docker.com/r/valum/valum/. It is really convenient, but I understand if you prefer to host it along existing apps.

We could have a Vagrantfile in this repository which replicate the hosting environment with Valum installed.

Contributor

arteymix commented Nov 16, 2016

Valum can be installed system-wide or used locally with Meson. I don't have Ubuntu packages yet, but I think people at elementary are good at this. If that works well, it will be ready for a final release.

I maintain a Docker image for deployment based on the latest Ubuntu: https://hub.docker.com/r/valum/valum/. It is really convenient, but I understand if you prefer to host it along existing apps.

We could have a Vagrantfile in this repository which replicate the hosting environment with Valum installed.

@arteymix

This comment has been minimized.

Show comment
Hide comment
@arteymix

arteymix Nov 18, 2016

Contributor

It seems like the VM is using an old verison of libsoup, but Valum is has some fallback mode for that. In more recent versions, it will only warn for providing the interface property and use GIO SocketAddress API.

It's up to you all if you accept the branding for using Valum. I really don't mind removing it.

Contributor

arteymix commented Nov 18, 2016

It seems like the VM is using an old verison of libsoup, but Valum is has some fallback mode for that. In more recent versions, it will only warn for providing the interface property and use GIO SocketAddress API.

It's up to you all if you accept the branding for using Valum. I really don't mind removing it.

Minor improvements for the backend
Explicitly use 'accept' to avoid serving static files with 'text/html'
content type.

Use a single 'serve_from_file' middleware for all static resources.

Use a single try-catch for both navigation and content templates.
@arteymix

This comment has been minimized.

Show comment
Hide comment
@arteymix

arteymix Nov 18, 2016

Contributor

It should be fully backward-compatible now.

There's probably a bug on the client side and lack of understanding this aspect prevents me from fixing it:

jquery.min.js:124 Uncaught DOMException: Failed to execute 'appendChild' on 'Node': Nodes of type '#document' may not be inserted inside nodes of type '#document-fragment'.(…)

I'll finish caching with ETag and look upstream with Meson to get the 0.36 working on Travis CI.

Contributor

arteymix commented Nov 18, 2016

It should be fully backward-compatible now.

There's probably a bug on the client side and lack of understanding this aspect prevents me from fixing it:

jquery.min.js:124 Uncaught DOMException: Failed to execute 'appendChild' on 'Node': Nodes of type '#document' may not be inserted inside nodes of type '#document-fragment'.(…)

I'll finish caching with ETag and look upstream with Meson to get the 0.36 working on Travis CI.

@arteymix

This comment has been minimized.

Show comment
Hide comment
@arteymix

arteymix Nov 18, 2016

Contributor

Feel free to test it and give feedback. I'm pretty much done here. For potential contributors, I wrote a lot of documentation: http://docs.valum-framework.org/en/latest/

Contributor

arteymix commented Nov 18, 2016

Feel free to test it and give feedback. I'm pretty much done here. For potential contributors, I wrote a lot of documentation: http://docs.valum-framework.org/en/latest/

@arteymix

This comment has been minimized.

Show comment
Hide comment
@arteymix

arteymix Nov 18, 2016

Contributor

There's still search and tooltip to reimplement :)

Contributor

arteymix commented Nov 18, 2016

There's still search and tooltip to reimplement :)

@arteymix

This comment has been minimized.

Show comment
Hide comment
@arteymix

arteymix Nov 18, 2016

Contributor

@lewisgoddard How is the Sphinx database deployed? I think we should add this to the Vagrantfile.

Contributor

arteymix commented Nov 18, 2016

@lewisgoddard How is the Sphinx database deployed? I think we should add this to the Vagrantfile.

@arteymix

This comment has been minimized.

Show comment
Hide comment
@arteymix

arteymix Nov 18, 2016

Contributor

Ok, there's a layout for reimplementing /tooltip.php and /search.php features with libgda.

Contributor

arteymix commented Nov 18, 2016

Ok, there's a layout for reimplementing /tooltip.php and /search.php features with libgda.

@lewisgoddard

This comment has been minimized.

Show comment
Hide comment
@lewisgoddard

lewisgoddard Nov 19, 2016

Member

@arteymix with great difficulty? @btkostner has the details on Sphinx deployment.

Member

lewisgoddard commented Nov 19, 2016

@arteymix with great difficulty? @btkostner has the details on Sphinx deployment.

@btkostner

This comment has been minimized.

Show comment
Hide comment
@btkostner

btkostner Nov 19, 2016

Member

@arteymix I tried to get down the basics down in the README.md when I figured it out. It shouldn't be too hard. The port doesn't need to be accessable from the outside, and you can just use a standard mysql interface to connect to it and query. Let me know if I can be of help!

EDIT: Just looked at the last comment. It seems you have all of the mysql stuff figured out already.

Member

btkostner commented Nov 19, 2016

@arteymix I tried to get down the basics down in the README.md when I figured it out. It shouldn't be too hard. The port doesn't need to be accessable from the outside, and you can just use a standard mysql interface to connect to it and query. Let me know if I can be of help!

EDIT: Just looked at the last comment. It seems you have all of the mysql stuff figured out already.

@btkostner

This comment has been minimized.

Show comment
Hide comment
@btkostner

btkostner Nov 19, 2016

Member

I don't have push permission to that branch (or I just suck at git which is a very valid reason), so here are some of the changes I made to the Vagrantfile.

  • change "ubuntu/trusty64" to "ubuntu/xenial64" for more up to date packages
  • add sphinxsearch and xsltproc to line 70 apt-get install
  • Add the following lines after the valum build to generate the docs
    # Build valadoc files
    cd /home/vagrant/valadoc-org
    make
    make app
    make build-docs
  • Add the following lines after the docs generate to setup and run sphinx in daemon mode
    # Setup sphinx search
    mkdir -p /home/vagrant/valadoc-org/sphinx/storage
    ./configgen ./valadoc.org
    indexer --config ./sphinx.conf --all
    searchd --config ./sphinx.conf

This will work for when you first create it, but it won't start up after restarts.

Member

btkostner commented Nov 19, 2016

I don't have push permission to that branch (or I just suck at git which is a very valid reason), so here are some of the changes I made to the Vagrantfile.

  • change "ubuntu/trusty64" to "ubuntu/xenial64" for more up to date packages
  • add sphinxsearch and xsltproc to line 70 apt-get install
  • Add the following lines after the valum build to generate the docs
    # Build valadoc files
    cd /home/vagrant/valadoc-org
    make
    make app
    make build-docs
  • Add the following lines after the docs generate to setup and run sphinx in daemon mode
    # Setup sphinx search
    mkdir -p /home/vagrant/valadoc-org/sphinx/storage
    ./configgen ./valadoc.org
    indexer --config ./sphinx.conf --all
    searchd --config ./sphinx.conf

This will work for when you first create it, but it won't start up after restarts.

@benwaffle

This comment has been minimized.

Show comment
Hide comment
@benwaffle

benwaffle Nov 19, 2016

Contributor

Since we're not using php anymore, it doesn't make sense for the URLs to be /search.php and /tooltip.php

Contributor

benwaffle commented Nov 19, 2016

Since we're not using php anymore, it doesn't make sense for the URLs to be /search.php and /tooltip.php

@arteymix

This comment has been minimized.

Show comment
Hide comment
@arteymix

arteymix Nov 30, 2016

Contributor

It now builds docs and generates Sphinx index on provisioning. I will probably finish reimplementing search and tooltip feature by this week-end :)

Contributor

arteymix commented Nov 30, 2016

It now builds docs and generates Sphinx index on provisioning. I will probably finish reimplementing search and tooltip feature by this week-end :)

@arteymix

This comment has been minimized.

Show comment
Hide comment
@arteymix

arteymix Nov 30, 2016

Contributor

@benwaffle I agree too, but I think it would be nice to have something drop-in first, because I don't really want to touch the frontend.

Then we'll have plenty of room for major improvements. I would really like to push forward some content negotiation for serving our docs in JSON and XML as well.

Contributor

arteymix commented Nov 30, 2016

@benwaffle I agree too, but I think it would be nice to have something drop-in first, because I don't really want to touch the frontend.

Then we'll have plenty of room for major improvements. I would really like to push forward some content negotiation for serving our docs in JSON and XML as well.

Makefile
@@ -140,6 +145,6 @@ test-examples: valadoc-example-tester
#
serve: default build-docs
- FWD_SEARCH=1 FWD_TOOLTIP=1 php -S localhost:7777 -t ./valadoc.org ./valadoc.org/router.php

This comment has been minimized.

@KubaKaszycki

KubaKaszycki Dec 1, 2016

Nice - removing PHP. Wonder why it was still there 😆

@KubaKaszycki

KubaKaszycki Dec 1, 2016

Nice - removing PHP. Wonder why it was still there 😆

Update to Valum 0.3.1
It brings some internal improvement for the static resource delivery
middlewares we use for assets.
@arteymix

This comment has been minimized.

Show comment
Hide comment
@arteymix

arteymix Dec 6, 2016

Contributor

It seems like libgda is using an unsupported feature from Sphinx MySQL frontend. I'll investigate and try to patch it.

Contributor

arteymix commented Dec 6, 2016

It seems like libgda is using an unsupported feature from Sphinx MySQL frontend. I'll investigate and try to patch it.

@arteymix

This comment has been minimized.

Show comment
Hide comment
@arteymix

arteymix Dec 29, 2016

Contributor

I opened a ticket on bugzilla for the MySQL issue: https://bugzilla.gnome.org/show_bug.cgi?id=776581

Contributor

arteymix commented Dec 29, 2016

I opened a ticket on bugzilla for the MySQL issue: https://bugzilla.gnome.org/show_bug.cgi?id=776581

arteymix added some commits Feb 21, 2017

backend: Fix handling of missing documents
 - use the error message instead of a generic one
 - check if 'DocCacheEntry.contents' is 'null' instead
 - throw the error upstream if the main navigation is missing
backend: Reduce the error-handling logic with the cache
Basic error propagation is now provided by 'g_lru_cache_get' so errors
from 'load_contents' can now be handled properly.
backend: Don't restore the query as it would display results twice
Instead, we have the regular overview of packages when we land on search
results.
backend: Retrieve the current package from a hidden search input
This implicitly fix the retrieval of 'package' for non-API pages as they
won't set its corresponding context entry.
backend: Use Valum '0.3.7' with support for cancelled requests
It should make development more bearable and reduce significantly the
number of warnings for NULL I/O streams.
@arteymix

This comment has been minimized.

Show comment
Hide comment
@arteymix

arteymix Feb 24, 2017

Contributor

I added a hidden input for the search page to include the package, which is less hackish than checking the current URL.

It's pretty much all functional with fixed @ in symbols.

We have to be careful when merging because the main template is embedded in the code and I think there's been some good UI work already.

Contributor

arteymix commented Feb 24, 2017

I added a hidden input for the search page to include the package, which is less hackish than checking the current URL.

It's pretty much all functional with fixed @ in symbols.

We have to be careful when merging because the main template is embedded in the code and I think there's been some good UI work already.

@benwaffle

This comment has been minimized.

Show comment
Hide comment
@benwaffle

benwaffle Jun 18, 2017

Contributor

Can we add glrucache in-tree? Or maybe use meson for the vala code and just have valum, compose, and glrucache be subprojects?

Contributor

benwaffle commented Jun 18, 2017

Can we add glrucache in-tree? Or maybe use meson for the vala code and just have valum, compose, and glrucache be subprojects?

@benwaffle

This comment has been minimized.

Show comment
Hide comment
@benwaffle

benwaffle Jun 18, 2017

Contributor

Also I'd like a dockerfile for development so I don't have to have sphinx installed. Vagrant uses virtualbox which is kinda heavy

Contributor

benwaffle commented Jun 18, 2017

Also I'd like a dockerfile for development so I don't have to have sphinx installed. Vagrant uses virtualbox which is kinda heavy

@arteymix

This comment has been minimized.

Show comment
Hide comment
@arteymix

arteymix Jun 18, 2017

Contributor

We can add a Dockerfile. I think that if we stick on the same distribution, we can have a common provision script and avoid duplicating all the initialization.

I did not introduce Meson to keep this as a drop-in replacement. If we do this, then it will make a lot of sense to use the compatible dependencies via subprojects.

EDIT: I submitted this patch https://github.com/chergert/glrucache/pull/6 so we can use it properly and I have a local version that current work using Meson

Contributor

arteymix commented Jun 18, 2017

We can add a Dockerfile. I think that if we stick on the same distribution, we can have a common provision script and avoid duplicating all the initialization.

I did not introduce Meson to keep this as a drop-in replacement. If we do this, then it will make a lot of sense to use the compatible dependencies via subprojects.

EDIT: I submitted this patch https://github.com/chergert/glrucache/pull/6 so we can use it properly and I have a local version that current work using Meson

@benwaffle

This comment has been minimized.

Show comment
Hide comment
@benwaffle

benwaffle Oct 25, 2017

Contributor

Lets use meson subprojects for valum/vsgi/glrucache like this: https://github.com/benwaffle/pulse-flow/blob/master/subprojects/libgtkflow.wrap

Contributor

benwaffle commented Oct 25, 2017

Lets use meson subprojects for valum/vsgi/glrucache like this: https://github.com/benwaffle/pulse-flow/blob/master/subprojects/libgtkflow.wrap

@desiderantes

This comment has been minimized.

Show comment
Hide comment
@desiderantes

desiderantes Jun 24, 2018

Contributor

Sad to see this incomplete:( Is it still viable?

Contributor

desiderantes commented Jun 24, 2018

Sad to see this incomplete:( Is it still viable?

@ZanderBrown

This comment has been minimized.

Show comment
Hide comment
@ZanderBrown

ZanderBrown Jun 24, 2018

Contributor

Hopefully this can be finished, so much could be done with a JSON/XML API built on top of this

Contributor

ZanderBrown commented Jun 24, 2018

Hopefully this can be finished, so much could be done with a JSON/XML API built on top of this

@arteymix

This comment has been minimized.

Show comment
Hide comment
@arteymix

arteymix Jun 24, 2018

Contributor

@ZanderBrown I'm really busy right now with my research project and I truly hope I could put some time on this side-project by the end of the summer.

If anyone want to take the lead, I'll be glad to coordinate the efforts.

Contributor

arteymix commented Jun 24, 2018

@ZanderBrown I'm really busy right now with my research project and I truly hope I could put some time on this side-project by the end of the summer.

If anyone want to take the lead, I'll be glad to coordinate the efforts.

@ZanderBrown

This comment has been minimized.

Show comment
Hide comment
@ZanderBrown

ZanderBrown Jun 25, 2018

Contributor

If we have a clear TODO i'm willing to try & find time for a couple items

Contributor

ZanderBrown commented Jun 25, 2018

If we have a clear TODO i'm willing to try & find time for a couple items

@arteymix

This comment has been minimized.

Show comment
Hide comment
@arteymix

arteymix Jun 25, 2018

Contributor

I'll upload a fully working Vagrant setup tomorrow, fix the merge conflicts and assess the remaining work to be done into a proper TODO list.

I think it's okay to use the coming VSGI 1.0 and Valum 0.4 which are already near stabilization.

Contributor

arteymix commented Jun 25, 2018

I'll upload a fully working Vagrant setup tomorrow, fix the merge conflicts and assess the remaining work to be done into a proper TODO list.

I think it's okay to use the coming VSGI 1.0 and Valum 0.4 which are already near stabilization.

@benwaffle

This comment has been minimized.

Show comment
Hide comment
@benwaffle

benwaffle Jun 25, 2018

Contributor

https://github.com/eustasy/vala-docs.org - this generates an XML file containing all of the info needed. Would be good if the valum backend used that

Contributor

benwaffle commented Jun 25, 2018

https://github.com/eustasy/vala-docs.org - this generates an XML file containing all of the info needed. Would be good if the valum backend used that

@btkostner

This comment has been minimized.

Show comment
Hide comment
@btkostner

btkostner Jul 12, 2018

Member

The more and more I think about this, I think the best route would be to have a Valum backend that uses the XML file generator to create an API only backend server.

Then having a frontend project that consumes the API just like every other client (builder, docs, etc.)

This keeps the backend all in vala. Easy to edit and maintain for vala devs. Also gives us more freedom with the frontend to offer some cool things like an pwa, and offline support. Maybe even an easier time including guides and less API-y like documentation.

EDIT:
Also, if we build the backend with Docker, we could potentially have mutliple backend servers up for different vala versions (if we really wanted that.). 0-36.api.valadoc.org

Member

btkostner commented Jul 12, 2018

The more and more I think about this, I think the best route would be to have a Valum backend that uses the XML file generator to create an API only backend server.

Then having a frontend project that consumes the API just like every other client (builder, docs, etc.)

This keeps the backend all in vala. Easy to edit and maintain for vala devs. Also gives us more freedom with the frontend to offer some cool things like an pwa, and offline support. Maybe even an easier time including guides and less API-y like documentation.

EDIT:
Also, if we build the backend with Docker, we could potentially have mutliple backend servers up for different vala versions (if we really wanted that.). 0-36.api.valadoc.org

@arteymix

This comment has been minimized.

Show comment
Hide comment
@arteymix

arteymix Jul 12, 2018

Contributor

For Docker, I intended to setup a docker-compose environment to run the sphinx search engine and the HTTP backend separately. I don't know if that is consistent with your future hosting goals, but that could be a nice-to-have setup for scaling and targeting other hosting providers.

The Vagrant setup works for testing purposes, but I'm afraid I cannot put more time and efforts on this side project for now.

Contributor

arteymix commented Jul 12, 2018

For Docker, I intended to setup a docker-compose environment to run the sphinx search engine and the HTTP backend separately. I don't know if that is consistent with your future hosting goals, but that could be a nice-to-have setup for scaling and targeting other hosting providers.

The Vagrant setup works for testing purposes, but I'm afraid I cannot put more time and efforts on this side project for now.

@btkostner

This comment has been minimized.

Show comment
Hide comment
@btkostner

btkostner Jul 12, 2018

Member

Eventually down the line I would love to have everything just running in docker containers on kubernetes, but it's going to be a while before that happens. I was thinking as far as deployment goes, having CI create a docker image that contains all of the compiled vala code and docs as that time. That way if you need to go back in time and look at previous docs for some reason, it's just changing the docker image version. This also makes it a lot easier to deploy and pull down locally if you need.

Member

btkostner commented Jul 12, 2018

Eventually down the line I would love to have everything just running in docker containers on kubernetes, but it's going to be a while before that happens. I was thinking as far as deployment goes, having CI create a docker image that contains all of the compiled vala code and docs as that time. That way if you need to go back in time and look at previous docs for some reason, it's just changing the docker image version. This also makes it a lot easier to deploy and pull down locally if you need.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment