Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[COST-4515] Upgrade to Django 4.1 #4853

Merged
merged 19 commits into from
Feb 2, 2024
Merged

Conversation

djnakabaale
Copy link
Contributor

@djnakabaale djnakabaale commented Jan 4, 2024

Jira Ticket

COST-4515

Description

This change will upgrade Django version from 4.0.x to 4.1.x.

Testing

  1. Checkout Branch
  2. Restart Koku
  3. Hit endpoint or launch shell
    1. You should see ...
  4. Do more things...

Notes

...

@djnakabaale djnakabaale added the smoke-tests pr_check will build the image and run minimal required smokes label Jan 4, 2024
Copy link

codecov bot commented Jan 30, 2024

Codecov Report

Merging #4853 (0748230) into cost-2231-django-4 (c382291) will increase coverage by 0.0%.
The diff coverage is 82.6%.

Additional details and impacted files
@@                Coverage Diff                 @@
##           cost-2231-django-4   #4853   +/-   ##
==================================================
  Coverage                93.9%   94.0%           
==================================================
  Files                     370     370           
  Lines                   30573   30565    -8     
  Branches                 3648    3645    -3     
==================================================
+ Hits                    28716   28718    +2     
+ Misses                   1197    1187   -10     
  Partials                  660     660           

koku/api/iam/models.py Outdated Show resolved Hide resolved
djnakabaale and others added 13 commits February 2, 2024 10:05
This assert statement will always fail since the template schema does not yet
exists. This test will only pass if another test runs before this test creates
the template schema.
Do not define a custom sequence object since that is no longer required with the
move to identity columns.

Co-authored-by: Michael Skarbek <mskarbek@redhat.com>
Co-authored-by: David Nakabaale <dnakabaa@redhat.com>
We may not end up wanting this to be so automatic, but this is a good way to
handle it if needed.

The only problem is getting the hash is a bit clunky since the contents of
clone_schema.sql do not exactly match the function definition once it is stored
in the database. I added the hash output to logs in order to make updating the
hash easier.
@samdoran samdoran force-pushed the cost-4515-django-4.1 branch 2 times, most recently from f3c643f to bcc7e39 Compare February 2, 2024 16:58
@djnakabaale djnakabaale marked this pull request as ready for review February 2, 2024 17:51
@djnakabaale djnakabaale requested review from a team as code owners February 2, 2024 17:51
samdoran and others added 5 commits February 2, 2024 13:09
Do not define a custom sequence object since that is no longer required with the
move to identity columns.

Co-authored-by: Michael Skarbek <mskarbek@redhat.com>
Co-authored-by: David Nakabaale <dnakabaa@redhat.com>
Add correct copyright and license information to management command.

Create project level licenses to reflect the use of multiple licenses within
the project.
- Use SQL composition rather than format strings to protect against SQL injection
  attacks. The risk is quite low, but it’s best to just do the right thing.
- Reorganize the code to avoid a nested function definiton and an unnecessary
  global variable that only needed to be local.
Copy link

sonarcloud bot commented Feb 2, 2024

Quality Gate Failed Quality Gate failed

Failed conditions

1 Security Hotspot
24.7% Coverage on New Code (required ≥ 85%)

See analysis details on SonarCloud

@djnakabaale djnakabaale merged commit ee25d85 into cost-2231-django-4 Feb 2, 2024
10 of 12 checks passed
@djnakabaale djnakabaale deleted the cost-4515-django-4.1 branch February 2, 2024 18:23
@djnakabaale djnakabaale mentioned this pull request Feb 2, 2024
3 tasks
@djnakabaale djnakabaale restored the cost-4515-django-4.1 branch February 2, 2024 20:02
samdoran added a commit that referenced this pull request Feb 9, 2024
* upgrade to Django 4.1

* Update deprecated ‘warn’ method

* Remove deprecated test case method

* Remove deprecated Series.bool method

* Correct docstring and class attribute name

* Cleanup file handle

* Fix flaky test

This assert statement will always fail since the template schema does not yet
exists. This test will only pass if another test runs before this test creates
the template schema.

* Update clone_schema.sql

Do not define a custom sequence object since that is no longer required with the
move to identity columns.

Co-authored-by: Michael Skarbek <mskarbek@redhat.com>
Co-authored-by: David Nakabaale <dnakabaa@redhat.com>

* Update clone_schema SQL function when necessary

We may not end up wanting this to be so automatic, but this is a good way to
handle it if needed.

The only problem is getting the hash is a bit clunky since the contents of
clone_schema.sql do not exactly match the function definition once it is stored
in the database. I added the hash output to logs in order to make updating the
hash easier.

* Avoid unnecessary globals

* update comments in clone_schema func

* Update clone_schema hash

* add management cmd to migrate serial cols

* Update clone_schema.sql

Do not define a custom sequence object since that is no longer required with the
move to identity columns.

Co-authored-by: Michael Skarbek <mskarbek@redhat.com>
Co-authored-by: David Nakabaale <dnakabaa@redhat.com>

* Update license information

Add correct copyright and license information to management command.

Create project level licenses to reflect the use of multiple licenses within
the project.

* Improvements to serial to identity management command

- Use SQL composition rather than format strings to protect against SQL injection
  attacks. The risk is quite low, but it’s best to just do the right thing.
- Reorganize the code to avoid a nested function definiton and an unnecessary
  global variable that only needed to be local.

* Do not update clone_schema function automatically

* Add migration to update clone_schema function

* fixup! Improvements to serial to identity management command

---------

Co-authored-by: Sam Doran <sdoran@redhat.com>
Co-authored-by: Michael Skarbek <mskarbek@redhat.com>
samdoran added a commit that referenced this pull request Feb 12, 2024
* upgrade to Django 4.1

* Update deprecated ‘warn’ method

* Remove deprecated test case method

* Remove deprecated Series.bool method

* Correct docstring and class attribute name

* Cleanup file handle

* Fix flaky test

This assert statement will always fail since the template schema does not yet
exists. This test will only pass if another test runs before this test creates
the template schema.

* Update clone_schema.sql

Do not define a custom sequence object since that is no longer required with the
move to identity columns.

Co-authored-by: Michael Skarbek <mskarbek@redhat.com>
Co-authored-by: David Nakabaale <dnakabaa@redhat.com>

* Update clone_schema SQL function when necessary

We may not end up wanting this to be so automatic, but this is a good way to
handle it if needed.

The only problem is getting the hash is a bit clunky since the contents of
clone_schema.sql do not exactly match the function definition once it is stored
in the database. I added the hash output to logs in order to make updating the
hash easier.

* Avoid unnecessary globals

* update comments in clone_schema func

* Update clone_schema hash

* add management cmd to migrate serial cols

* Update clone_schema.sql

Do not define a custom sequence object since that is no longer required with the
move to identity columns.

Co-authored-by: Michael Skarbek <mskarbek@redhat.com>
Co-authored-by: David Nakabaale <dnakabaa@redhat.com>

* Update license information

Add correct copyright and license information to management command.

Create project level licenses to reflect the use of multiple licenses within
the project.

* Improvements to serial to identity management command

- Use SQL composition rather than format strings to protect against SQL injection
  attacks. The risk is quite low, but it’s best to just do the right thing.
- Reorganize the code to avoid a nested function definiton and an unnecessary
  global variable that only needed to be local.

* Do not update clone_schema function automatically

* Add migration to update clone_schema function

* fixup! Improvements to serial to identity management command

---------

Co-authored-by: Sam Doran <sdoran@redhat.com>
Co-authored-by: Michael Skarbek <mskarbek@redhat.com>
samdoran added a commit that referenced this pull request Feb 13, 2024
* upgrade to Django 4.1

* Update deprecated ‘warn’ method

* Remove deprecated test case method

* Remove deprecated Series.bool method

* Correct docstring and class attribute name

* Cleanup file handle

* Fix flaky test

This assert statement will always fail since the template schema does not yet
exists. This test will only pass if another test runs before this test creates
the template schema.

* Update clone_schema.sql

Do not define a custom sequence object since that is no longer required with the
move to identity columns.

Co-authored-by: Michael Skarbek <mskarbek@redhat.com>
Co-authored-by: David Nakabaale <dnakabaa@redhat.com>

* Update clone_schema SQL function when necessary

We may not end up wanting this to be so automatic, but this is a good way to
handle it if needed.

The only problem is getting the hash is a bit clunky since the contents of
clone_schema.sql do not exactly match the function definition once it is stored
in the database. I added the hash output to logs in order to make updating the
hash easier.

* Avoid unnecessary globals

* update comments in clone_schema func

* Update clone_schema hash

* add management cmd to migrate serial cols

* Update clone_schema.sql

Do not define a custom sequence object since that is no longer required with the
move to identity columns.

Co-authored-by: Michael Skarbek <mskarbek@redhat.com>
Co-authored-by: David Nakabaale <dnakabaa@redhat.com>

* Update license information

Add correct copyright and license information to management command.

Create project level licenses to reflect the use of multiple licenses within
the project.

* Improvements to serial to identity management command

- Use SQL composition rather than format strings to protect against SQL injection
  attacks. The risk is quite low, but it’s best to just do the right thing.
- Reorganize the code to avoid a nested function definiton and an unnecessary
  global variable that only needed to be local.

* Do not update clone_schema function automatically

* Add migration to update clone_schema function

* fixup! Improvements to serial to identity management command

---------

Co-authored-by: Sam Doran <sdoran@redhat.com>
Co-authored-by: Michael Skarbek <mskarbek@redhat.com>
samdoran added a commit that referenced this pull request Feb 13, 2024
* upgrade to Django 4.1

* Update deprecated ‘warn’ method

* Remove deprecated test case method

* Remove deprecated Series.bool method

* Correct docstring and class attribute name

* Cleanup file handle

* Fix flaky test

This assert statement will always fail since the template schema does not yet
exists. This test will only pass if another test runs before this test creates
the template schema.

* Update clone_schema.sql

Do not define a custom sequence object since that is no longer required with the
move to identity columns.

Co-authored-by: Michael Skarbek <mskarbek@redhat.com>
Co-authored-by: David Nakabaale <dnakabaa@redhat.com>

* Update clone_schema SQL function when necessary

We may not end up wanting this to be so automatic, but this is a good way to
handle it if needed.

The only problem is getting the hash is a bit clunky since the contents of
clone_schema.sql do not exactly match the function definition once it is stored
in the database. I added the hash output to logs in order to make updating the
hash easier.

* Avoid unnecessary globals

* update comments in clone_schema func

* Update clone_schema hash

* add management cmd to migrate serial cols

* Update clone_schema.sql

Do not define a custom sequence object since that is no longer required with the
move to identity columns.

Co-authored-by: Michael Skarbek <mskarbek@redhat.com>
Co-authored-by: David Nakabaale <dnakabaa@redhat.com>

* Update license information

Add correct copyright and license information to management command.

Create project level licenses to reflect the use of multiple licenses within
the project.

* Improvements to serial to identity management command

- Use SQL composition rather than format strings to protect against SQL injection
  attacks. The risk is quite low, but it’s best to just do the right thing.
- Reorganize the code to avoid a nested function definiton and an unnecessary
  global variable that only needed to be local.

* Do not update clone_schema function automatically

* Add migration to update clone_schema function

* fixup! Improvements to serial to identity management command

---------

Co-authored-by: Sam Doran <sdoran@redhat.com>
Co-authored-by: Michael Skarbek <mskarbek@redhat.com>
samdoran added a commit that referenced this pull request Feb 14, 2024
* upgrade to Django 4.1

* Update deprecated ‘warn’ method

* Remove deprecated test case method

* Remove deprecated Series.bool method

* Correct docstring and class attribute name

* Cleanup file handle

* Fix flaky test

This assert statement will always fail since the template schema does not yet
exists. This test will only pass if another test runs before this test creates
the template schema.

* Update clone_schema.sql

Do not define a custom sequence object since that is no longer required with the
move to identity columns.

Co-authored-by: Michael Skarbek <mskarbek@redhat.com>
Co-authored-by: David Nakabaale <dnakabaa@redhat.com>

* Update clone_schema SQL function when necessary

We may not end up wanting this to be so automatic, but this is a good way to
handle it if needed.

The only problem is getting the hash is a bit clunky since the contents of
clone_schema.sql do not exactly match the function definition once it is stored
in the database. I added the hash output to logs in order to make updating the
hash easier.

* Avoid unnecessary globals

* update comments in clone_schema func

* Update clone_schema hash

* add management cmd to migrate serial cols

* Update clone_schema.sql

Do not define a custom sequence object since that is no longer required with the
move to identity columns.

Co-authored-by: Michael Skarbek <mskarbek@redhat.com>
Co-authored-by: David Nakabaale <dnakabaa@redhat.com>

* Update license information

Add correct copyright and license information to management command.

Create project level licenses to reflect the use of multiple licenses within
the project.

* Improvements to serial to identity management command

- Use SQL composition rather than format strings to protect against SQL injection
  attacks. The risk is quite low, but it’s best to just do the right thing.
- Reorganize the code to avoid a nested function definiton and an unnecessary
  global variable that only needed to be local.

* Do not update clone_schema function automatically

* Add migration to update clone_schema function

* fixup! Improvements to serial to identity management command

---------

Co-authored-by: Sam Doran <sdoran@redhat.com>
Co-authored-by: Michael Skarbek <mskarbek@redhat.com>
samdoran added a commit that referenced this pull request Feb 15, 2024
* Update deprecated ‘warn’ method
* Remove deprecated test case method
* Remove deprecated Series.bool method
* Correct docstring and class attribute name
* Cleanup file handle
* Fix flaky test

  This assert statement will always fail since the template schema does not yet
  exists. This test will only pass if another test runs before this test creates
  the template schema.

* Update clone_schema.sql

  Do not define a custom sequence object since that is no longer required with the
  move to identity columns.

  Do not define a custom sequence object since that is no longer required with the
  move to identity columns.

* Avoid unnecessary globals
* Add management command to migrate serial ID columns to identity

* Update license information

  Add correct copyright and license information to management command.

  Create project level licenses to reflect the use of multiple licenses within
  the project.

* Improvements to serial to identity management command

  Use SQL composition rather than format strings to protect against SQL injection
  attacks. The risk is quite low, but it’s best to just do the right thing.
  Reorganize the code to avoid a nested function definiton and an unnecessary
  global variable that only needed to be local.

* Add migration to update clone_schema function

---------

Co-authored-by: Sam Doran <sdoran@redhat.com>
Co-authored-by: Michael Skarbek <mskarbek@redhat.com>
Co-authored-by: David Nakabaale <dnakabaa@redhat.com>
samdoran added a commit that referenced this pull request Feb 16, 2024
* Update deprecated ‘warn’ method
* Remove deprecated test case method
* Remove deprecated Series.bool method
* Correct docstring and class attribute name
* Cleanup file handle
* Fix flaky test

  This assert statement will always fail since the template schema does not yet
  exists. This test will only pass if another test runs before this test creates
  the template schema.

* Update clone_schema.sql

  Do not define a custom sequence object since that is no longer required with the
  move to identity columns.

  Do not define a custom sequence object since that is no longer required with the
  move to identity columns.

* Avoid unnecessary globals
* Add management command to migrate serial ID columns to identity

* Update license information

  Add correct copyright and license information to management command.

  Create project level licenses to reflect the use of multiple licenses within
  the project.

* Improvements to serial to identity management command

  Use SQL composition rather than format strings to protect against SQL injection
  attacks. The risk is quite low, but it’s best to just do the right thing.
  Reorganize the code to avoid a nested function definiton and an unnecessary
  global variable that only needed to be local.

* Add migration to update clone_schema function

---------

Co-authored-by: Sam Doran <sdoran@redhat.com>
Co-authored-by: Michael Skarbek <mskarbek@redhat.com>
Co-authored-by: David Nakabaale <dnakabaa@redhat.com>
samdoran pushed a commit that referenced this pull request Feb 19, 2024
The upgrade from Django 3.2 → 4.2 was done in stages

Upgrade to Django 4.0 (#4839)
  - Remove gettext alias
  - Mocks for get_response required by middlewaremixin
  - Use method_decorator to apply neevr_cache on class method
  - Method set_context deprecated in DRF 3.12 in favor of requires_context
  - User Django RedisCache and remove django_redis library


Upgrade to Django 4.1 (#4853)
  - Update deprecated ‘warn’ method
  - Remove deprecated test case method
  - Remove deprecated Series.bool method
  - Correct docstring and class attribute name
  - Cleanup file handle
  - Fix flaky test
  
    This assert statement will always fail since the template schema does not yet
    exists. This test will only pass if another test runs before this test creates
    the template schema.
  
  - Update clone_schema.sql
  
    Do not define a custom sequence object since that is no longer required with the
    move to identity columns.
  
    Do not define a custom sequence object since that is no longer required with the
    move to identity columns.
  
  - Avoid unnecessary globals
  - Add management command to migrate serial ID columns to identity
  
  - Update license information
  
    Add correct copyright and license information to management command.
  
    Create project level licenses to reflect the use of multiple licenses within
    the project.
  
  - Improvements to serial to identity management command
  
    Use SQL composition rather than format strings to protect against SQL injection
    attacks. The risk is quite low, but it’s best to just do the right thing.
    Reorganize the code to avoid a nested function definiton and an unnecessary
    global variable that only needed to be local.
  
  - Add migration to update clone_schema function


Upgrade to Django 4.2 (#4893)
  - Clean up unittest  test_execute_query_with_group_by_tag_and_limit
  - Move Window after aggregation in group_by annotation
  
    Since the Window is used only to provide row numbers and there is no additional
    filtering done on the data in the Window, moving it after the ArrayAgg fixes the
    error encountered in Django 4.2 and returns the same data in the same order.
  
  - Check correct log level in test
    This works in Python 3.9 due to a bug in unittest.
  
  - Use test database name from settings instead of hard coding it
  - Use consistent name for test database
  - Correct patching in test
  - Update clone_schema function in managament command
  - Stop using deprecated .bool() method
  
    https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.bool.html
  
  - Explicitly set the test database name
  
    Currently we are relying on Django’s internal test setup to add a `test_` prefix
    to the default database name. This works fine most of the time but it’s better
    to be explicit.
  
  - Use method call instead of operator
  
    This prevents tripping up flake8 and the need for an ignore or parenthesis
  
  - Reduce to a final boolean value
  
    https://pandas.pydata.org/pandas-docs/stable/user_guide/basics.html#boolean-reductions
  
  - Remove migration that updates clone_schema SQL function
  
    Handling this exclusively with the migrate_serial_to_identity_columns
    management command
  
  - Move to STORAGES from deprecated options
  - Remove deprecated L10_N setting

---------

Co-authored-by: Michael Skarbek <mskarbek@redhat.com>
Co-authored-by: David Nakabaale <dnakabaa@redhat.com>
@maskarb maskarb deleted the cost-4515-django-4.1 branch July 12, 2024 18:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
smoke-tests pr_check will build the image and run minimal required smokes smokes-required
Projects
None yet
3 participants