Migration of web apps to Pyramid framework
Thoughts and considerations for moving the opentree web apps (synth-tree viewer and curation tool) from web2py to the Pyramid framework. Pyramid is now our preferred web framework for new projects, so we've discussed moving these apps as well. What's involved here, and what decisions need to be made? Are there components (e.g. for web-form handling) that are preferred by other OpenTree devs, or do I have free rein to pick these?
[2021 UPDATE: Seeing no other comments here, I'm going ahead with some choices below, based on prior work and my best judgement. =jimA=]
Pserve is apparently a capable production server, at least up to some (unspecified) high volume of requests. WSGI will insulate us from most server differences, so we can probably stick with that unless OT devs have a strong preference for Apache + mod_wsgi.
Should we consider WSGI middleware (authentication/authorisation, session stuff, caching, filtering)? Or stay within Pyramid?
[WSGI middleware is not as well-regarded these days. I'll try to stay within Pyramid and add-ons.]
Probably not, but review our URL conventions (and rules for permissions vs. "containment") to help decide if this might be appropriate. [We're still not using containment in any important way, so URL dispatch (w/ a few complicated cases) looks like the right answer.]
Similarly, what about using route factories to fetch a context object (resource), vs. just using inline sql queries?
This might make sense, esp. if we have an elaborate dance to build up a composite resource. Or if we have complex per-object ACLs to add.
Can SQLAlchemy be used with our git-based storage? Or extended in some way, so that we can take advantage of its transaction boundaries?
--
How can we handle complex file-upload scenarios with Pyramid? I get the impression this doesn't exactly come out of the box.
On closer examination, it's not very complicated. Here are some recipes and notes:
- https://docs.pylonsproject.org/projects/pyramid-cookbook/en/latest/static_assets/uploading-files.html
- https://docs.pylonsproject.org/projects/pyramid-cookbook/en/latest/forms/file_uploads.html
- http://flask.pocoo.org/docs/0.12/patterns/fileuploads/
Probably using view predicates (based on file extension or the Accepts HTTP header. Here are some good ideas:
- https://cecilphillip.com/content-negotiation-with-the-pyramid-web-framework/
- https://docs.pylonsproject.org/projects/pyramid/en/latest/narr/viewconfig.html#accept-header-content-negotiation
The latter (newer?) approach simply checks the Accept header directly in @view_config.
Here are some options, including redis:
- https://docs.pylonsproject.org/projects/pyramid/en/latest/narr/sessions.html#using-alternate-session-factories
- https://pypi.org/project/pyramid_redis_sessions/
It looks like ws_wrapper considered using pyramid_beaker (and the underlying Beaker) for sessions, and possibly caching? Maybe @bredelings can shed some light on this.
- https://docs.pylonsproject.org/projects/pyramid/en/latest/narr/events.html#creating-your-own-events
- https://docs.pylonsproject.org/projects/pyramid/en/latest/api/events.html#event-types
See https://docs.pylonsproject.org/projects/pyramid/en/latest/narr/subrequest.html
package (Pyramid app)? Or just refer to these in the filesystem?
I suppose the "Pyramid way" is that the application (and repo) is a python package...
When reviewing our existing Pyramid projects, only Chameleon makes an appearance, but it appears to be unused. So I'm guessing we have not established a precedent.
After looking at samples and design differences between some popular (and Pyramid-friendly) templating systems, I think Jinja2 looks like a solid choice, mainly for its
- decent performance and debugging
- appealing notation (similar to Django templates, familiar to many developers)
- allowance for very basic in-template logic (but does not encourage this!)
Lots of ideas here on setup for auto-rotating logs, etc.
NOTE that we can globally log exceptions to stdout or a file (best for production):
Pyramid's config file has many options to fine-tune logging:
Here's an initial approach:
- sensible global settings (msg format, etc)
- module-specific loggers as needed
- to segregate messages to a special log file
- to use alternate msg formats
- for extra DEBUG-level logging, as needed
- create ad-hoc loggers during dev/testing? (These should inherit sensible behavior/destination from global settings.)
Reminder: There's some shared code, shared auth/sessions, etc. Maybe we can correct the shared code aspects, at least, by making a proper/separate module. (Should all apps share a repo?)