Search - Banish the ALN Columns #3378
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Terraform plan for dev No changes. Your infrastructure matches the configuration.✅ Plan applied in Deploy to Development and Management Environment #545 |
|
Terraform plan for meta No changes. Your infrastructure matches the configuration.✅ Plan applied in Deploy to Development and Management Environment #545 |
tadhg-ohiggins
approved these changes
Feb 8, 2024
|
Minimum allowed coverage is Generated by 🐒 cobertura-action against fe78579 |
purvinptl
pushed a commit
that referenced
this pull request
Feb 12, 2024
* Banish the ALN columns, for now... * Return the right results. Disable column tests.
purvinptl
pushed a commit
that referenced
this pull request
Feb 12, 2024
* Banish the ALN columns, for now... * Return the right results. Disable column tests.
purvinptl
pushed a commit
that referenced
this pull request
Feb 12, 2024
* Banish the ALN columns, for now... * Return the right results. Disable column tests.
purvinptl
pushed a commit
that referenced
this pull request
Feb 12, 2024
* Banish the ALN columns, for now... * Return the right results. Disable column tests.
github-merge-queue bot
pushed a commit
that referenced
this pull request
Mar 4, 2024
* Tribal api add_remove function and request access function * Removed request_file_access since it was implemented in create_function.sql in admin api * bugfixes; grant permissions for tribal access tables * 3288 cannot convert string to integer (#3290) * Code refactoring * #3288 More refactoring * Added logic to handle empty finding count, empty program total, empty cluster total and empty amount expend * Ensure calculated value are returned as string * Erroring on "GSA_MIGRATION" in emails when not migrating (#3275) * Checking for GSA_MIGRATION when not migrating * Adding unit tests * Fixing test name * Fixing test names * Reverting is_gsa_migration param * Trailing comma * Moving check to validate_general_information_json * Removing print * Lint * Setting is_gsa_migration to False for api serializer * Using getattr to fix tests * Adding test_views.py test * Lint * Not using getattr * Back up prod media files every two hours. (#3239) * Copy Updates Re: Our Lack of Email Notifications (#3229) * Copy re email notifs: audit access form * Copy re email notifs: lock for certification * Copy re email notifs: auditor certification * Keep wording consistent * add prefix-ALN note to program name validation (#3230) * Adds columns to FederalAwards, Findings in summary sheets (#3296) * Adds columns to FederalAwards, Findings Adds "_aln" to FederalAward Adds "_federal_agency_prefix", "_federal_award_extension" and "_aln" to Finding. This makes it possible for resolution officials to easily find the reference numbers on their awards without having to manually flip back-and-forth between tabs, which is error prone. * Linting. * Speed up search queries (#3283) * Trying to optimize queries. Result of cowork with JP. Handing off. * This is a step towards improving queries. The first part, intersecting things, is a good way to filter down. ALNs could still use improvement. * Updating Moving towards implementing ALN sorting. * Handing off... * THIS WORKS * Checkpoint * Checkpoint from end of day * Forgot the constants * This... makes it work. However, like the rest of the code, it assumes we have indicies. So. We may roll this back. * Backing out the ForeignKey * Adding the API back, reverting summary reports * Commenting out possible indexes for future * Undoing debug toolbar * Linting! * "Finding all ALN" -> "Finding other ALN" * Test fixes/updates, minor finding_all_aln tweaks * is_public = True OR is_public = False, not AND * Really, really working v2.3.2 * Actually really working 3.0 * Adds ALN Y/N based on ALNs, not just agency numbers. * Handoff. * Default to a 2023 search over getting everything * Readd annotate_findings_two, to compare approaches * A little linting * Test updates for 2023-default behavior * Final push * Logging changes * Working things. * A blank line, for fun * Linting. * Eliminating incorrect comment. * Don't create dict as default argument. * Clearer tribal test * Undo RAM grab in Postgres local container. --------- Co-authored-by: James Person <jperson1@umbc.edu> Co-authored-by: Tadhg O'Higgins <2626258+tadhg-ohiggins@users.noreply.github.com> * TEST SITE Banner (#3261) * Test site banner, test implementation * Linting to appease the lint gods. * Sometimes we copy/paste too fast. * Mshd/handle reports without notes (#3300) * Updated Schema to Allow GSA migration keyword in NotesToSefa coversheet * Prevent use of GSA migration keyword outside of migration process * Added logic to generate Note records with GSA migration keyword only when no record found for 2018/17/16 * Fixed failing test * Uncomment Schedule Block (#3308) * 3309 unexpected loan balance (#3311) * #3309 Updated logic to default zero loan balance to empty when loan flag is false * #3309 Updated test cases * 3312 discrepancy in the number of passthrough names compared to passthrough ids (#3316) * Renamed function to improve readability * #3312 Updated logic to handle cardinality mismatch between passthrough names and ids * #3312 Adding test * 3315 missing reference numbers in audit findings text records (#3319) * Updated logic to handle missing reference numbers * Updated schema * Census migration: Adding CAP text missing ref unit tests (#3321) * Adding cap text tests * Generating wb * Generator fix * Lint * Cleaning up generator * Lint * "!=" is not the same thing as "is not" (#3326) * Context for 500s, Resolve Pre-cert Report 500s (#3333) * 500 errors should have (some limited) context. * Pre-cert reports should not append bonus fields. * Pre-cert reports should not append bonus fields. * Updated dev and preview instance memories (#3334) * django admin tooling for dissem (#3342) * Omit Underscored Fields from Pre-certification Reports (#3340) * 500 errors should have (some limited) context. * Pre-cert reports should not append bonus fields. * Pre-cert reports should not append bonus fields. * Omit underscored fields from column names. * Added has_tribal_data_access function * fix tribal filtering clause, fix migration conflict * Bumped api versions for dissemination/api and support/admin_api to 1_1_0 * linter * fix add/remove privs for tribal access * add cypress test for tribal API access * update testing docs * Github Action Version Updates (#3310) * Bump create-or-update-comment * Bump Zap * Bump repository-dispatch * Bump create-pull-request * Bump auto-approve-action * Increase production RAM to 6G. (#3376) * Search - Banish the ALN Columns (#3378) * Banish the ALN columns, for now... * Return the right results. Disable column tests. * Capture search parameters in NR (#3382) * try add search params as NR attributes * make them strings * maybe all of the search fields now? * audit_years is a special case because they're ints * maybe spell it right though * lint * Search - Default FederalAward Results to Empty QuerySet (#3384) * r_full_alns should default to an empty queryset, not None. * Now with a test! * Update Django And Cryptography package (#3380) * bump requirements * python lib updates * Use only disseminated data in the public API (#3381) * Fixes gross error in API. The `OR` conjunction in every view was causing a combinatoric explosion on every single API query, destroying performance as the DB grew. This fixes that problem. * Fixes the general view Uses only disseminated data instead of the internal table. THis broke with historical data, because not all environments have a historical singleauditchecklist (at this time). * Fixing accidental removal Leave the additional_ueis there. * Updating version on dropped tables * Search - Banish the ALN Columns (#3378) * Banish the ALN columns, for now... * Return the right results. Disable column tests. * Search - Default FederalAward Results to Empty QuerySet (#3384) * r_full_alns should default to an empty queryset, not None. * Now with a test! * Use only disseminated data in the public API (#3381) * Fixes gross error in API. The `OR` conjunction in every view was causing a combinatoric explosion on every single API query, destroying performance as the DB grew. This fixes that problem. * Fixes the general view Uses only disseminated data instead of the internal table. THis broke with historical data, because not all environments have a historical singleauditchecklist (at this time). * Fixing accidental removal Leave the additional_ueis there. * Search - Banish the ALN Columns (#3378) * Banish the ALN columns, for now... * Return the right results. Disable column tests. * Search - Default FederalAward Results to Empty QuerySet (#3384) * r_full_alns should default to an empty queryset, not None. * Now with a test! * Use only disseminated data in the public API (#3381) * Fixes gross error in API. The `OR` conjunction in every view was causing a combinatoric explosion on every single API query, destroying performance as the DB grew. This fixes that problem. * Fixes the general view Uses only disseminated data instead of the internal table. THis broke with historical data, because not all environments have a historical singleauditchecklist (at this time). * Fixing accidental removal Leave the additional_ueis there. * Search - Banish the ALN Columns (#3378) * Banish the ALN columns, for now... * Return the right results. Disable column tests. * Search - Default FederalAward Results to Empty QuerySet (#3384) * r_full_alns should default to an empty queryset, not None. * Now with a test! * Removed the old api version api_v1_0_3 * Made suggested changes during code review * api versioning fixes * keep 1_0_3 the default * also docker-compose-web * rollback changes to api 1_0_3 * newlines * Dropping the redundant drop.sql file since we have drop_schema.sql * Similar changes to api_v1_0_3 with redundant drop schema. * resolve migration conflict * Renamed drop.sql to drop_schema.sql to match other changes * Making the hasAgency const logic clear. * Refactored p_report_id to report_id for consistent naming. * Dropped the redundant drop.sql file * specify API version in cypress tests * add api_v1_1_0 to tf --------- Co-authored-by: Tim Ballard <1425377+timoballard@users.noreply.github.com> Co-authored-by: Hassan D. M. Sambo <hassandeme.mamasambo@gsa.gov> Co-authored-by: Phil Dominguez <142051477+phildominguez-gsa@users.noreply.github.com> Co-authored-by: Tadhg O'Higgins <2626258+tadhg-ohiggins@users.noreply.github.com> Co-authored-by: James Person <james.person@gsa.gov> Co-authored-by: Dan Swick <2365503+danswick@users.noreply.github.com> Co-authored-by: Matthew Jadud <matthew.jadud@gsa.gov> Co-authored-by: James Person <jperson1@umbc.edu> Co-authored-by: Alex Steel <130377221+asteel-gsa@users.noreply.github.com>
eazapata23
pushed a commit
that referenced
this pull request
Mar 5, 2024
* Tribal api add_remove function and request access function * Removed request_file_access since it was implemented in create_function.sql in admin api * bugfixes; grant permissions for tribal access tables * 3288 cannot convert string to integer (#3290) * Code refactoring * #3288 More refactoring * Added logic to handle empty finding count, empty program total, empty cluster total and empty amount expend * Ensure calculated value are returned as string * Erroring on "GSA_MIGRATION" in emails when not migrating (#3275) * Checking for GSA_MIGRATION when not migrating * Adding unit tests * Fixing test name * Fixing test names * Reverting is_gsa_migration param * Trailing comma * Moving check to validate_general_information_json * Removing print * Lint * Setting is_gsa_migration to False for api serializer * Using getattr to fix tests * Adding test_views.py test * Lint * Not using getattr * Back up prod media files every two hours. (#3239) * Copy Updates Re: Our Lack of Email Notifications (#3229) * Copy re email notifs: audit access form * Copy re email notifs: lock for certification * Copy re email notifs: auditor certification * Keep wording consistent * add prefix-ALN note to program name validation (#3230) * Adds columns to FederalAwards, Findings in summary sheets (#3296) * Adds columns to FederalAwards, Findings Adds "_aln" to FederalAward Adds "_federal_agency_prefix", "_federal_award_extension" and "_aln" to Finding. This makes it possible for resolution officials to easily find the reference numbers on their awards without having to manually flip back-and-forth between tabs, which is error prone. * Linting. * Speed up search queries (#3283) * Trying to optimize queries. Result of cowork with JP. Handing off. * This is a step towards improving queries. The first part, intersecting things, is a good way to filter down. ALNs could still use improvement. * Updating Moving towards implementing ALN sorting. * Handing off... * THIS WORKS * Checkpoint * Checkpoint from end of day * Forgot the constants * This... makes it work. However, like the rest of the code, it assumes we have indicies. So. We may roll this back. * Backing out the ForeignKey * Adding the API back, reverting summary reports * Commenting out possible indexes for future * Undoing debug toolbar * Linting! * "Finding all ALN" -> "Finding other ALN" * Test fixes/updates, minor finding_all_aln tweaks * is_public = True OR is_public = False, not AND * Really, really working v2.3.2 * Actually really working 3.0 * Adds ALN Y/N based on ALNs, not just agency numbers. * Handoff. * Default to a 2023 search over getting everything * Readd annotate_findings_two, to compare approaches * A little linting * Test updates for 2023-default behavior * Final push * Logging changes * Working things. * A blank line, for fun * Linting. * Eliminating incorrect comment. * Don't create dict as default argument. * Clearer tribal test * Undo RAM grab in Postgres local container. --------- Co-authored-by: James Person <jperson1@umbc.edu> Co-authored-by: Tadhg O'Higgins <2626258+tadhg-ohiggins@users.noreply.github.com> * TEST SITE Banner (#3261) * Test site banner, test implementation * Linting to appease the lint gods. * Sometimes we copy/paste too fast. * Mshd/handle reports without notes (#3300) * Updated Schema to Allow GSA migration keyword in NotesToSefa coversheet * Prevent use of GSA migration keyword outside of migration process * Added logic to generate Note records with GSA migration keyword only when no record found for 2018/17/16 * Fixed failing test * Uncomment Schedule Block (#3308) * 3309 unexpected loan balance (#3311) * #3309 Updated logic to default zero loan balance to empty when loan flag is false * #3309 Updated test cases * 3312 discrepancy in the number of passthrough names compared to passthrough ids (#3316) * Renamed function to improve readability * #3312 Updated logic to handle cardinality mismatch between passthrough names and ids * #3312 Adding test * 3315 missing reference numbers in audit findings text records (#3319) * Updated logic to handle missing reference numbers * Updated schema * Census migration: Adding CAP text missing ref unit tests (#3321) * Adding cap text tests * Generating wb * Generator fix * Lint * Cleaning up generator * Lint * "!=" is not the same thing as "is not" (#3326) * Context for 500s, Resolve Pre-cert Report 500s (#3333) * 500 errors should have (some limited) context. * Pre-cert reports should not append bonus fields. * Pre-cert reports should not append bonus fields. * Updated dev and preview instance memories (#3334) * django admin tooling for dissem (#3342) * Omit Underscored Fields from Pre-certification Reports (#3340) * 500 errors should have (some limited) context. * Pre-cert reports should not append bonus fields. * Pre-cert reports should not append bonus fields. * Omit underscored fields from column names. * Added has_tribal_data_access function * fix tribal filtering clause, fix migration conflict * Bumped api versions for dissemination/api and support/admin_api to 1_1_0 * linter * fix add/remove privs for tribal access * add cypress test for tribal API access * update testing docs * Github Action Version Updates (#3310) * Bump create-or-update-comment * Bump Zap * Bump repository-dispatch * Bump create-pull-request * Bump auto-approve-action * Increase production RAM to 6G. (#3376) * Search - Banish the ALN Columns (#3378) * Banish the ALN columns, for now... * Return the right results. Disable column tests. * Capture search parameters in NR (#3382) * try add search params as NR attributes * make them strings * maybe all of the search fields now? * audit_years is a special case because they're ints * maybe spell it right though * lint * Search - Default FederalAward Results to Empty QuerySet (#3384) * r_full_alns should default to an empty queryset, not None. * Now with a test! * Update Django And Cryptography package (#3380) * bump requirements * python lib updates * Use only disseminated data in the public API (#3381) * Fixes gross error in API. The `OR` conjunction in every view was causing a combinatoric explosion on every single API query, destroying performance as the DB grew. This fixes that problem. * Fixes the general view Uses only disseminated data instead of the internal table. THis broke with historical data, because not all environments have a historical singleauditchecklist (at this time). * Fixing accidental removal Leave the additional_ueis there. * Updating version on dropped tables * Search - Banish the ALN Columns (#3378) * Banish the ALN columns, for now... * Return the right results. Disable column tests. * Search - Default FederalAward Results to Empty QuerySet (#3384) * r_full_alns should default to an empty queryset, not None. * Now with a test! * Use only disseminated data in the public API (#3381) * Fixes gross error in API. The `OR` conjunction in every view was causing a combinatoric explosion on every single API query, destroying performance as the DB grew. This fixes that problem. * Fixes the general view Uses only disseminated data instead of the internal table. THis broke with historical data, because not all environments have a historical singleauditchecklist (at this time). * Fixing accidental removal Leave the additional_ueis there. * Search - Banish the ALN Columns (#3378) * Banish the ALN columns, for now... * Return the right results. Disable column tests. * Search - Default FederalAward Results to Empty QuerySet (#3384) * r_full_alns should default to an empty queryset, not None. * Now with a test! * Use only disseminated data in the public API (#3381) * Fixes gross error in API. The `OR` conjunction in every view was causing a combinatoric explosion on every single API query, destroying performance as the DB grew. This fixes that problem. * Fixes the general view Uses only disseminated data instead of the internal table. THis broke with historical data, because not all environments have a historical singleauditchecklist (at this time). * Fixing accidental removal Leave the additional_ueis there. * Search - Banish the ALN Columns (#3378) * Banish the ALN columns, for now... * Return the right results. Disable column tests. * Search - Default FederalAward Results to Empty QuerySet (#3384) * r_full_alns should default to an empty queryset, not None. * Now with a test! * Removed the old api version api_v1_0_3 * Made suggested changes during code review * api versioning fixes * keep 1_0_3 the default * also docker-compose-web * rollback changes to api 1_0_3 * newlines * Dropping the redundant drop.sql file since we have drop_schema.sql * Similar changes to api_v1_0_3 with redundant drop schema. * resolve migration conflict * Renamed drop.sql to drop_schema.sql to match other changes * Making the hasAgency const logic clear. * Refactored p_report_id to report_id for consistent naming. * Dropped the redundant drop.sql file * specify API version in cypress tests * add api_v1_1_0 to tf --------- Co-authored-by: Tim Ballard <1425377+timoballard@users.noreply.github.com> Co-authored-by: Hassan D. M. Sambo <hassandeme.mamasambo@gsa.gov> Co-authored-by: Phil Dominguez <142051477+phildominguez-gsa@users.noreply.github.com> Co-authored-by: Tadhg O'Higgins <2626258+tadhg-ohiggins@users.noreply.github.com> Co-authored-by: James Person <james.person@gsa.gov> Co-authored-by: Dan Swick <2365503+danswick@users.noreply.github.com> Co-authored-by: Matthew Jadud <matthew.jadud@gsa.gov> Co-authored-by: James Person <jperson1@umbc.edu> Co-authored-by: Alex Steel <130377221+asteel-gsa@users.noreply.github.com> Signed-off-by: Edward Zapata <99770283+eazapata23@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Search - Banish the ALN Columns
For now...
Changes:
Disable the "Finding my/all ALN" columns in the search queries. They are too expensive for the amount of data we have, and will need to be revisited shortly.
PR checklist: submitters
maininto your branch shortly before creating the PR. (You should also be mergingmaininto your branch regularly during development.)git status | grep migrations. If there are any results, you probably need to add them to the branch for the PR. Your PR should have only one new migration file for each of the component apps, except in rare circumstances; you may need to delete some and re-runpython manage.py makemigrationsto reduce the number to one. (Also, unless in exceptional circumstances, your PR should not delete any migration files.)PR checklist: reviewers
make docker-clean; make docker-first-run && docker compose up; then rundocker compose exec web /bin/bash -c "python manage.py test"The larger the PR, the stricter we should be about these points.