Dashboard: Dockerize the drag-n-drop webapp

[felipesanches]

Currently the drag-n-drop webapp hosted at http://fontbakery.appspot.com/ does not run FontForge checks (or any other checks that require usage of the subprocess module) due to secutiry restrictions on Google App Engine.

If we migrate the webapp to Google Container Engine, we'll be able to run these additional checks, improving the coverage of the reports geenrated by this webapp.

[felipesanches]

I've been chatting to a friend and he suggested that maybe the webapp could remain hosted at GAppEngine, and simply dispatch jobs on the same containers used for our dashboard system infrastructure. Sounds like a neat approach.

[opsxcq]

We can keep the GAE application, for job orchestration it's possible to use google cloud pub/sub, it have a fair price ($0.06 per GB).

An use case example, the user drag-n-drop:

• User drag a font to fontbakery.appspot.com
• GAE Application put a message in the job queue with the font and generate an ID, return it to JS in the user browser so it can start pooling for the results.
• Worker get the message from the queue, process it, and add the result to another queue,
• GAE Application receive the message and store the results permanently in the database identified by the ID
• User get the job result for generated ID and display it

Another interesting use case that this solution can solve, a git push:

• User configure his github account to use webhooks to trigger specialized endpoint on http://fontbakery.appspot.com/
• GAE Application create a job when that endpoint is triggered (a different type, maybe "git", or another type to differentiate from job request above, create a different field or queue for that) and generate an ID for the job
• Worker checkout the font repository, and run the tests, and post the results for that job in the queue
• GAE Application receive the message and store the results permanently in the database identified by the ID

About this second use case, where to display the result ?

If we choose not to use google cloud pub/sub, is possible to implement a simple queue system with GAE, and make the worker simply use a pooling for get the jobs.

Containers can run on Google Container Engine or Google Compute Engine, some questions to help to choose:

• What are the satisfactory requirements to run a test ? (Memory, processor and disk capacity)
• How many requests/jobs would run in a day ?
• How much it will cost on both platforms to run those tests with a satisfactory performance ?

[felipesanches]

Google Fonts has got a bit more than 8 hundred font families. In a simple first prototype of the dockerized dashboard (which is not really the topic of this issue, but #1223 instead) we could leave out webhooks and simply re-run all checks on the full set of families every N days (or whenever there's some manual stats update trigger requested by a dashboard user).

A better implementation employing webhooks could reduce the amount of testsuite reruns.

But for the drag-n-drop app, the problem seems much simpler. The goal of this issue is simply to have the current webapp ported to docker to benefit from the additional checks that rely on the python subprocess module.

[m4rc1e]

Hey guys,

I've used Docker on Digital Ocean for gfregression, site.

Reason for my own virtual server was so I can have free reign of my dependencies. If I want to jump to aws, it is no problem.

[davelab6]

I'm assuming that as a g project we can get free credit on g cloud and d.o./AWS are persona non grata ;) G pubsub or firebase or whatever is fine to consider but (a) is it it fine for Felipe? (Who is committed to software freedom in his daily practice :) and (b) rethinkdb or other similar nosql databases provide pubsub directly.

…

On Feb 17, 2017 12:21 PM, "Marc Foley" ***@***.***> wrote: Hey guys, I've used Docker on Digital Ocean for gfregression <https://github.com/m4rc1e/gfregression>, site <http://45.55.138.144>. Reason for my own virtual server was so I can have free reign of my dependencies. If I want to jump to aws, it is no problem. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#1230 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAP9y-hmupiLTkaf97uBJNqZ7KXUU9e_ks5rdWaZgaJpZM4MDs2N> .

[davelab6]

On Feb 17, 2017 4:00 AM, "Felipe Corrêa da Silva Sanches" < notifications@github.com> wrote: Google Fonts has got a bit more than 8 hundred font families. In a dump first prototype of the dockerized dashboard (which is not really the topic of this issue, but #1223 <#1223> instead) we could leave out webhooks and simply re-run all checks on the full set of families every N days (or whenever there's some manual stats update trigger requested by a dashboard user). Leave both alone for now, I think. Instead I suggest to focus on drag and drop, then on what's in production, and then doing what's in production - and then that all in parallel. Looking back, then that across every commit to the repo, to graph the results over time. This is actually not that important, looking forward is more urgent :) Looking forward, then those two things can be combined, so that prod and drag'n'drop set up for comparison (ie, with gfregression integrated) such that we can say, prod has X detected problems and the version dropped has Y problems, and the impact has been compared and contrasted, so we have a high degree of confidence that if the dropped fonts replace the current fonts in prod, they will fix stuff without introducing regressions or any unexpected changes. And also forward, I expect that FB itself will be updated much more frequently than the fonts... So, maybe the code modularization you wanted to do could help make parallel reruns even faster, if only the new/updated tests are run when those are updated... A better implementation employing webhooks could reduce the amount of testsuite reruns. Hooking up to stuff in github is secondary imho. What I think is primary is, the ability to launch bug free new families, and to compare what is in production now to a local copy dropped in. But for the drag-n-drop app, the problem seems much simpler. The goal of this issue is simply to have the current webapp ported to docker to benefit from the additional checks that rely on the python subprocess module Right, from my POV, the gae version was meant to be fast to deliver, even though it was a "half" measure and inherently limited, and for be is now deprecated. A "full" Docker based fb drag and drop is on the critical path to a full solution for sure, and likely a very next step :) 🛫

[felipesanches]

Now that the dockerized dashboard is running on GKE, I think the natural next step is to port the drag and drop feature into that and then deprecate the App Engine instance.

[davelab6]

Agree

…

On Mar 9, 2017 11:27 PM, "Felipe Corrêa da Silva Sanches" < ***@***.***> wrote: Now that the dockerized dashboard is running on GKE, I think the natural next step is to port the drag and drop feature into that and then deprecate the App Engine instance. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#1230 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAP9y8ykmFsiwl5AUvCzXIhlf7swPbUEks5rkNFAgaJpZM4MDs2N> .

[felipesanches]

Migrating to fontbakery-dashboard issue tracker per #1383