Backend version updates #605

rm03 · 2024-01-22T05:17:31Z

This PR upgrades Django from version 3.2 to 5.0 (should be able to try out GeneratedField in #601!) and sets the Python version to 3.11 from 3 (tried upgrading to 3.12 but unfortunately some packages still need to add support for it, so it's better to wait) along with some other related upgrades.

Additionally, it replaces Black, Flake8, and Isort with Ruff, which is a very fast linter and formatter. The translation from previous linting rules to Ruff's are close to but not exactly one-to-one, so we'll likely need to make some tweaks to get the configuration that we want.

gitguardian · 2024-01-22T05:17:35Z

⚠️ GitGuardian has uncovered 2 secrets following the scan of your pull request.

Please consider investigating the findings and remediating the incidents. Failure to do so may lead to compromising the associated services or software components.

🔎 Detected hardcoded secrets in your pull request

GitGuardian id	GitGuardian status	Secret	Commit	Filename
9335321	Triggered	Generic Private Key	`74d67e1`	nginx/certs/_wildcard.localhost.net-key.pem	View secret
9335321	Triggered	Generic Private Key	`f23ff2d`	nginx/certs/_wildcard.localhost.net-key.pem	View secret

🛠 Guidelines to remediate hardcoded secrets

Understand the implications of revoking this secret by investigating where it is used in your code.
Replace and store your secrets safely. Learn here the best practices.
Revoke and rotate these secrets.
If possible, rewrite git history. Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data.

To avoid such incidents in the future consider

following these best practices for managing and storing secrets including API keys and other credentials
install secret detection on pre-commit to catch secret before it leaves your machine and ease remediation.

^{🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.

Our GitHub checks need improvements? Share your feedbacks!}

rohangpta · 2024-01-23T19:28:38Z

backend/Pipfile

@@ -4,20 +4,21 @@ url = "https://pypi.org/simple"
 verify_ssl = true

 [dev-packages]
-black = "==19.10b0"
+black = "*"


Why is this still here? And why is it = "*"? (Same for flake8)

I might be mistaken, but I think the CI expects black and flake8 to be installed as part of the Django check, so until we migrate shared-actions to use ruff we should keep these here? I removed the version pins as I wasn't sure why they were necessary

Just updated shared-actions to use ruff, and got rid of black, flake8, and isort

rohangpta · 2024-01-23T19:42:32Z

backend/Pipfile

@@ -44,7 +45,7 @@ uvloop = {version = "*", sys_platform = "== 'linux'"}
 uvicorn = {extras = ["standard"], version = "*"}
 gunicorn = "*"
 httptools = "*"
-ics = "==0.7"
+ics = "*"


Can you just confirm we're not depending on a version pin here?

There was only one apparent issue with imports relating to the version pin which I fixed here

rohangpta · 2024-01-23T19:42:38Z

backend/Pipfile

@@ -34,7 +35,7 @@ qrcode = "*"
 python-dateutil = "*"
 psycopg2 = "*"
 django-simple-history = "*"
-channels = "<3"
+channels = "*"


Can you just confirm we're not depending on a version pin here?

I believe this was the only part of our code that needed to be updated, but I'll do further checks/tests locally to make sure unpinning the version doesnt cause any issues

rohangpta · 2024-01-23T19:43:02Z

backend/Pipfile

@@ -57,6 +58,8 @@ pandas = "*"
 drf-excel = "*"
 numpy = "*"
 coverage = "*"
+daphne = "*"


Can you explain why we need daphne and tatsu, and why tatsu needs a pin?

daphne is needed because channels.testing now requires it but doesn't install it as a subdependency (see this)

I think we should be able to get rid of the tatsu pin. I don't remember why I added it (could've been because they dropped support for Python 3.10, but this isn't relevant as we're using 3.11 anyway).

I remember why I added the tatsu pin now, it's because the latest Django base image uses Python 3.10. Will try and use an image with Python 3.11

rohangpta · 2024-01-23T19:43:38Z

.github/workflows/build-and-deploy.yaml

@@ -12,6 +12,7 @@ jobs:
    with:
      projectName: pennclubs
      path: backend
+      pythonVersion: 3.11-bookworm


What's this bookworm stuff?

bookworm is the latest stable Debian release and it seems that Docker has removed support for buster (which is two versions behind bookworm) with later versions of Python. Also if we don't specify the image it defaults to python3.8-buster which doesn't make sense now that we're using 3.11

rohangpta · 2024-01-23T19:55:16Z

Overall this looks good to me! Can you just clarify some of my comments, and tell me about all the local tests you've run with this?

We should also temporarily add in integration tests to just test out a CI run with them enabled. Very possible it could catch some unwanted behaviour or runtime exceptions due to deprecated functions that the tests don't otherwise catch.

A couple other things:

You should browse this page to make sure we aren't missing anything major: https://docs.djangoproject.com/en/5.0/internals/deprecation/
Looks like Postgres 11 is no longer supported -- EDIT: turns out this is not relevant, we upgraded to 15 last fall. Worth checking if any of our other deps are still compatible (really it's just redis, postgres, and maybe some websocket stuff like channels)

Here is what ChatGPT tells me to look out for (some of it is cruft but thought I'd paste here for you to read):

Django has undergone several changes from version 3.2 to 5.0, including new features and some backwards incompatible changes. Here's a summary of the key backwards incompatible changes in each major version from Django 3.2 to 5.0:

Django 3.2

Enhanced support for JSONB aggregation in PostgreSQL.
New database operations for PostgreSQL, like CreateCollation and RemoveCollation.
Additional attributes for Sitemap in django.contrib.sitemaps.
Improvements in error reporting, file uploads, and forms.
Updates to generic views and management commands.
Extensions to the migrations system and model functionalities.

Django 4.0

Introduction of Malay language support and changes in the DeleteView to use FormMixin.
Modifications in logging, management commands, and models, including the introduction of QuerySet.contains().
Changes in CSRF protection settings (CSRF_TRUSTED_ORIGINS).
Removal of support for older versions of PostGIS, GDAL, GEOS, PostgreSQL 9.6, and Oracle 12.2 and 18c.

Django 4.2

Improved support for locales in the makemessages command.
Enhancements in migrations, models, and requests/responses.
Updated utilities and validators.
Dropping support for MariaDB 10.3, MySQL 5.7, and PostgreSQL 11.
Changes in the database backend API, particularly for MySQL.

Django 5.0

Numerous feature additions across various components like asynchronous views, decorators, file storage, and internationalization.
Enhancements in error reporting, forms, migrations, models, pagination, signals, templates, tests, and validators.
Removal of support for older versions of GDAL and GEOS.
Changes in the database backend API, especially concerning the use of database functions as defaults and support for the DEFAULT keyword in INSERT queries.

Each of these versions introduced specific changes that might require adjustments in your Django projects, especially if you are directly upgrading from Django 3.2 to Django 5.0. It's important to review the detailed release notes and migration guides for each version to ensure a smooth upgrade process.

For detailed information, you can refer to the release notes for Django 5.0, Django 4.2, Django 4.0, and Django 3.2.

rm03 · 2024-02-12T21:23:34Z

For local testing I've only run the unit tests and backend and verified that there were no errors, but I agree that integration tests will definitely help us catch errors that the unit tests won't. I also looked at the Django deprecation page when making the upgrades and didn't notice any glaring incompatible changes, but will take a closer look to make sure I didn't miss anything

rm03 · 2024-02-20T01:53:52Z

Looks like integration tests are passing!

rohangpta · 2024-02-20T03:53:47Z

LGTM. A few guidelines around the merge

Please monitor sentry, DB util for 24+h
Squash and merge, so that revert is easy if needed
Things are likely to break, so be prepared.

rm03 added 9 commits January 6, 2024 15:22

first pass

b5a171b

configure ruff

f2e5528

pin black version

8f3dd99

update Python version in backend check, revert pin black version

86fd1ed

fix..?

363860d

switch to Python 3.11, fix tests

97bc979

fix dependency

3e00d64

Merge branch 'master' into version-upgrades

f23ff2d

lint

eec2172

rm03 requested a review from rohangpta January 22, 2024 05:21

rohangpta reviewed Jan 23, 2024

View reviewed changes

rm03 added 12 commits February 12, 2024 16:45

Merge branch 'master' into version-upgrades

74d67e1

fix channels

e458eaf

use ruff w/shared actions, remove black, flake8, isort

ae2dbd4

oops lol

ba8c264

this is embarrassing

b712858

use 3.11 base image

682b1d6

temporarily add integration tests

d8fe2fd

attempt to fix

0f8b271

fixed django base

c75d077

fix integration tests

f2df40b

fix csrf?

b9fde49

plz

58349c1

remove integration tests

659d62f

rohangpta self-requested a review February 20, 2024 03:53

rohangpta approved these changes Feb 20, 2024

View reviewed changes

rm03 merged commit 260a2b9 into master Feb 20, 2024
7 checks passed

rm03 deleted the version-upgrades branch February 20, 2024 04:34

aviupadhyayula mentioned this pull request Mar 10, 2024

Optimize assorted Django operations #627

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Backend version updates #605

Backend version updates #605

rm03 commented Jan 22, 2024

gitguardian bot commented Jan 22, 2024 •

edited

Loading

rohangpta Jan 23, 2024

rm03 Feb 12, 2024 •

edited

Loading

rm03 Feb 18, 2024

rohangpta Jan 23, 2024

rm03 Feb 12, 2024

rohangpta Jan 23, 2024

rm03 Feb 12, 2024

rohangpta Jan 23, 2024

rm03 Feb 12, 2024

rm03 Feb 18, 2024 •

edited

Loading

rohangpta Jan 23, 2024

rm03 Feb 12, 2024

rohangpta commented Jan 23, 2024 •

edited

Loading

rm03 commented Feb 12, 2024

rm03 commented Feb 20, 2024

rohangpta commented Feb 20, 2024

Backend version updates #605

Backend version updates #605

Conversation

rm03 commented Jan 22, 2024

gitguardian bot commented Jan 22, 2024 • edited Loading

⚠️ GitGuardian has uncovered 2 secrets following the scan of your pull request.

Choose a reason for hiding this comment

rm03 Feb 12, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rm03 Feb 18, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rohangpta commented Jan 23, 2024 • edited Loading

Django 3.2

Django 4.0

Django 4.2

Django 5.0

rm03 commented Feb 12, 2024

rm03 commented Feb 20, 2024

rohangpta commented Feb 20, 2024

gitguardian bot commented Jan 22, 2024 •

edited

Loading

rm03 Feb 12, 2024 •

edited

Loading

rm03 Feb 18, 2024 •

edited

Loading

rohangpta commented Jan 23, 2024 •

edited

Loading