Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

β → 🜚🏭 ③ #356

Merged
merged 59 commits into from May 31, 2021
Merged

β → 🜚🏭 ③ #356

merged 59 commits into from May 31, 2021

Conversation

HebaruSan
Copy link
Contributor

@HebaruSan HebaruSan commented May 22, 2021

Merge beta to master (alchemical symbol for gold, factory). Successor to: #301 #324
Includes: #340 #355

No particular hurry this time, but there's a lot here that will be very nice to have (speed-ups, profiling, CKAN integration fixes, better mod thumbnails).

User features:

Admin features:

Bug fixes:

I'm hoping we can do a production upgrade soon so #337 and the performance improvements can go live.

Upgrade considerations

Before upgrading:

  • In config.ini, set requests-per-profile to control how often we profile a request. I do not know the correct value for this because I do not know how much traffic the production server experiences. 1000, maybe?
  • In config.ini, set profile-dir to the path where profiling runs should be saved, for example on alpha and beta we are using /storage/sd-alpha/__profiling and /storage/sd-beta/__profiling
    • Think about maybe setting up a cron job to purge them, because otherwise they will multiply without limit. In alpha and beta we have this, which deletes data 36 hours old or older at 02:00 UTC:
      0 2 * * * find /storage/sd-alpha/__profiling/ -mtime 1.5 -delete
      
      Alpha has 125 MB in this folder as of this writing, but this is highly dependent on server load. Most of it is @DasSkelett's pinging of /kerbal-space-program/browse/top for monitoring and performance comparison every 3 minutes.

HebaruSan and others added 30 commits October 13, 2020 23:49
* We also register an error handler for `HTTPException` now. This one is invoked whenever we do something like `abort(403)`.
  The other one is invoked when there is a real exception raised. It does not rollback the database like the one for real exceptions,
  and it does not log to syslog. It has a separate template with new image.
* We register all those error handlers all the time, independent of `app.debug`. `app.debug` now only influences whether we
  show the 500 error template to the user, or launche the flask debugger. Whether we send an email only depends on whether the needed
  keys are set in the config, so it can be tested in dev envs too.
* Status code is now smaller than the message in the error templates.
* If a request accepts JSON and not HTML (or `*/*`), we return a JSONified exception, independent of request route (and exception type).
  For this to work, we send an explicit `Accept: application/json` header in XHR requests.
* Swap out 401s with 403s in many places.
This reduces the size of all HTML pages by around 33KB.
Which is aound 1/3 to 2/3 of their size.

Being in its own static file, this will be offloaded by the webserver,
and the reverse proxy and clients will be able to cache it.

This should reduce load on Flask/gunicorn by quite a lot.
HebaruSan and others added 15 commits May 7, 2021 11:59
Filter junk out of admin pages and make site 10x faster
Cache results of common database queries
Better messaging for descriptionless modpacks
prepare.sh now saves the package versions at the state of deployment of a new commit to alpha/beta/production
into a .frozen-requirements.txt file. As long as the HEAD commit doesn't change, every restart
will install the exact same package versions, to prevent silent, unwanted, breaking upgrades to some degree.

Also pin Flask to <2 and Jinja2 <3 until we tested it.
Prevent SqlAlchemy from upgrading to 2.0 when it is released,
since we use several deprecated functions that will be removed.
Remove some older pins that got added as a result of upstream bugs that should be resolved now.

Change Werkzeug profiler import from `werkzeug.contrib.profiler` to `werkzeug.middleware.profiler`,
it has been moved in 1.0.
Freeze dependencies on version they have at deployment
@HebaruSan HebaruSan added Type: Bug Type: Improvement Area: Backend Related to the Python code that runs inside gunicorn Priority: High Type: Feature Status: Ready Area: Frontend Related to HTML, JS, CSS, or other browser things labels May 22, 2021
@HebaruSan HebaruSan requested a review from DasSkelett May 22, 2021 13:27
Copy link
Member

@DasSkelett DasSkelett left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's go! Really looking forward to some of these changes.

@DasSkelett DasSkelett merged commit ebb54cb into master May 31, 2021
@HebaruSan HebaruSan mentioned this pull request Oct 8, 2021
@HebaruSan HebaruSan mentioned this pull request Jun 2, 2022
@HebaruSan HebaruSan mentioned this pull request Mar 4, 2023
@HebaruSan HebaruSan mentioned this pull request Apr 4, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Area: Backend Related to the Python code that runs inside gunicorn Area: Frontend Related to HTML, JS, CSS, or other browser things Priority: High Status: Ready Type: Bug Type: Feature Type: Improvement
Projects
None yet
2 participants