reimplement CSS using a renderer

[chadwhitacre]

Over on #3786 I want to make a dashboard.css that uses the same but separate SCSS wiring.

[chadwhitacre]

Ready for review @techtonik @rorepo @rohitpaulk @mattbk @kzisme et al.

[techtonik]

@whit537 how often does it regenerate CSS - on each request?

[chadwhitacre]

[H]ow often does it regenerate CSS - on each request?

Yup. That much hasn't changed: we've always generated CSS on each request to /assets/gratipay.css (and in production we hit this through a CDN in order to cache the generated CSS).

What this PR does is factor out the CSS-generation code from a simplate into a renderer, so that it can be used in multiple simplates (multiple *.css files).

[chadwhitacre]

@techtonik What other questions can I answer? :)

[webmaven]

Shouldn't gratipay.css.spt be renamed to gratipay.scss.spt?

[chadwhitacre]

Shouldn't gratipay.css.spt be renamed to gratipay.scss.spt?

No, because the css in gratipay.css.spt refers to the content type that an HTTP user agent is requesting, not the content type from which we render the content that we send to the HTTP user agent.

https://gratipay.com/assets/gratipay.css routes to gratipay.css.spt

[techtonik]

@whit537 I am -1 on regenerating SCSS on each request. Can we compile it only once on deploy (or watch and recompile on dev)?

[clone1018]

@techtonik Wouldn't it only generate once in production because of MaxCDN caching the request at their edge servers?

[techtonik]

@clone1018 even if so, it makes the system overcomplicated.

[clone1018]

@techtonik You "regenerate" html on every page request right? What's the difference?

[techtonik]

@clone1018 the difference is that HTML contain new information on each request so it new every time.

And you're right - future of web is API + client side web components that fetch only information that they need instead of that present hackish way of building some structure with the pieces of HTML strings.

[webmaven]

@techtonik:

even if so, it makes the system overcomplicated.

You're going to like this even less, then: We're probably going to have to go to a slightly more complex system, where the CSS is rendered on deploy, but put in a timestamped file and the hardcoded reference in the template is updated to point to the timestamped file, so the caches always know to find a new CSS file when the HTML changes.

This would guarantee that the CSS is cached whenever possible, but that stale CSS is never served.

We may then want to extend this to some other 'static' assets as well.

[Changaco]

In production assets are compiled during wireup.

[techtonik]

@webmaven by burying template compilation inside of Python code we are just increasing our tech/comp debts. I spend a weekend learning about buildpacks on Heroku and I liked that I've learned about the structure of web in gratipay/inside.gratipay.com#414

I think that the main reason why we have this hack is that gratipay.com is hosted on Heroku and Heroku can not execute the Makefile, and that is also the reason why our development setup is different from production. These are things that need to be fixed. It is devserver role to rebuild SCSS, and seeing that everybody will know about SCSS and automation reducing out competence debt.

[techtonik]

The Grunt/Gulp in JS world have a special "watch" task to rebuild the stuff (another thing that I've learned in the past month). Makefiles, of course, don't have such functionality. For Python Aspen could provide this ability, but as we move away from "Aspen as a web framework + devserver", https://pypi.python.org/pypi/livereload could be our solution.

[webmaven]

@Changaco yes, but the compiled assets re-use the same filenames (unless I am missing something), so caches might still serve stale assets.

@techtonik livereload basically causes the page to refresh (by having an injected JS script watch another port) if anything changes. This is only useful in local dev.

However, if we use a strategy where the HTML template itself is updated prior to (or during) deployment, to point to a timestamped or hashed version of the compiled asset (eg. 20151124123446785.main.css or d41d8cd98f00b204e9800998ecf8427e.main.css) as soon as the asset sources change and the asset is recompiled (or alternatively, that the template upon rendering inserts the versioned asset URLs), then as soon as the browser receives fresh HTML (whether from the cache, if it is cached at all, or from the server), the assets that the browser fetches (and the cache caches) will also be fresh, and because the URL for the asset will change if (and only if) a change is made to the source file, we can set the caching rules to cache these assets forever (or at least until they are flushed due to space constraints in the cache).

Gratipay used to use the site's version.txt for versioning static assets, but that changed to (AFAICT) rely on ETags headers exclusively: a91435c

Here is how Django does it: https://docs.djangoproject.com/en/1.8/howto/static-files/

A similar solution for Flask: https://github.com/ChrisTM/Flask-CacheBust

I would actually recommend using a framework-neutral library such as Fanstatic, BowerStatic, or WebAssets and integrating it with Aspen, simply to avoid reinventing this particular wheel over and over again.

[Changaco]

caches might still serve stale assets.

No, asset URLs are generated using the website.asset function and they include the etag in the querystring. URLs that don't contain an etag are only cached for 5 seconds.

avoid reinventing this particular wheel over and over again.

Sometimes it's faster to do it yourself, but maybe this will all get factored out with the move to Flask.

[webmaven]

@Changaco:

asset URLs are generated using the website.asset function and they include the etag in the querystring. URLs that don't contain an etag are only cached for 5 seconds.

Ah, including the ETag in the querystring will indeed have the same cache-busting behavior. Thanks for clearing that up.

[techtonik]

@webmaven given all that info, can you draw a picture of how Gratipay is currently compiled from different pieces into a web page that users see? If it will be SVG, I may be able then to show how it will look with a separate build phase.

[webmaven]

@techtonik:

given all that info, can you draw a picture of how Gratipay is currently compiled from different pieces into a web page that users see?

Er, perhaps. No promises, but please ping me if I don't deliver something in the next week or so.

If it will be SVG

If done, it will be SVG.

[techtonik]

Great! Next week I will most likely be offline, so ping will be unreliable.

[chadwhitacre]

Alright, so what's the status here? Can we merge this?

[webmaven]

So, I completely bailed on creating the diagram @techtonik requested (and it would still be handy to have), but given what @Changaco explained, I don't have any objection to merging, although I still think that a system that implemented render-on-deploy (or render-on-commit, or something similar) would be a bit more in line with current industry practices.

There is no real reason to wait for an HTTP request to trigger the rendering of the changed SCSS and only then cache it.

[chadwhitacre]

I've been experiencing "current industry practices" over on https://github.com/saxifrage/cityasacampus, and so far I think our system is better, though I haven't attempted or experienced live-reloading in either environment.

I don't have any objection to merging

May I invite you to click that big green "Merge pull request" button, @webmaven? ;-)

[techtonik]

@webmaven we will have to draw diagram anyway. I just haven't found any web based mindmapping software with open file format that can be stuffed into repository. Looks like I just to pump up my skills in Inkscape.

[chadwhitacre]

Thanks @webmaven! :-)