More Advanced Search Parameters & HTML Refactor #3623

jperson1 · 2024-04-09T20:42:59Z

More Advanced Search Parameters & HTML Refactor

Issues: #3576, #3622

Changes:

Keep search in one HTML file (search.html, instead of search.html + advanced.html). Give every filter its own file, and conditionally render them depending on which page you're on.
Add a few new filters, and change one:
- Fiscal year end month (basic + advanced)
- Type requirement (advanced)
- Entity type (basic + advanced)
- Passthrough name (advanced)
- Report ID (basic + advanced)
- UEI/EIN field searches the additional UEI/EIN tables in addition to general
Unit tests for the new filters

How to test:

Make sure you have some data loaded locally, and that your materialized view is up to date.
Switch to this branch and navigate to search.
Try the the new filters, and verify accuracy:
- Fiscal year end month. Searches on currently selected audit years. Most audits are either June or December. You can check a years results and verify that about half come back when using these months.
- Type requirement. The five most common are "P", "N", "L", "I", and "B". But, there are a huge number of valid options, and they can chain together. A good one is "ABCFGHIJLMN".
- Entity type. It doesn't seem like there are a lot of controls on this field, because "state" comes back with a lot of audits. But, it's what's the data shows.
- Passthrough name. Find a passthrough name locally (like "SDE"), and verify that the relevant report_ids come back in a search on it. Sometimes, there are a lot of results on one passthrough name.
- Report ID. Try a bunch!
- New UEI/EIN behavior. Find an audit with additional UEIs/EINs, and search on them to verify that the record now comes in, even if its general object does not represent these values.

Screenshots:

The new filters:

Most common type requirements in Census public data (~550 different combinations are represented):

PR checklist: submitters

PR checklist: reviewers

Pull the branch to your local environment and run make docker-clean; make docker-first-run && docker compose up; then run docker compose exec web /bin/bash -c "python manage.py test"
Manually test out the changes locally, or check this box to verify that it wasn’t applicable in this case.
Check that the PR has appropriate tests. Look out for changes in HTML/JS/JSON Schema logic that may need to be captured in Python tests even though the logic isn’t in Python.
Verify that no Git surgery is necessary at any point (such as during a merge party), or, if it was, repeat the testing after it’s finished.

The larger the PR, the stricter we should be about these points.

github-actions · 2024-04-09T21:02:22Z

File	Coverage		Missing
All files	`88%`	✅
api/serializers.py	`88%`	✅	`177-178` `183` `188`
api/test_views.py	`95%`	✅	`103`
api/uei.py	`88%`	✅	`87` `118-119` `163` `167-168`
api/views.py	`98%`	✅	`198-199` `337-338`
audit/forms.py	`60%`	❌	`31-38` `109-116`
audit/intake_to_dissemination.py	`88%`	✅	`57-62` `264` `308-316`
audit/mixins.py	`96%`	✅	`27`
audit/test_commands.py	`91%`	✅
audit/test_intakelib.py	`88%`	✅	`154-158`
audit/test_manage_submission_access_view.py	`98%`	✅	`15` `19`
audit/test_mixins.py	`90%`	✅	`159-160` `164-166` `254-255` `259-261`
audit/test_validators.py	`95%`	✅	`439` `443` `611-612` `851` `858` `865` `872` `1107-1108` `1139-1140` `1165-1170`
audit/test_views.py	`98%`	✅	`132`
audit/test_workbooks_should_fail.py	`88%`	✅	`58` `87-88` `92`
audit/test_workbooks_should_pass.py	`87%`	✅	`59` `74-76`
audit/utils.py	`91%`	✅	`33-35` `38`
audit/validators.py	`93%`	✅	`138` `190` `378-379` `394-395` `478-479` `581-585` `590-594` `610-619`
audit/cross_validation/additional_ueis.py	`93%`	✅	`33`
audit/cross_validation/check_award_ref_declaration.py	`90%`	✅
audit/cross_validation/check_award_reference_uniqueness.py	`93%`	✅
audit/cross_validation/check_certifying_contacts.py	`87%`	✅
audit/cross_validation/check_findings_count_consistency.py	`91%`	✅
audit/cross_validation/check_ref_number_in_cap.py	`90%`	✅
audit/cross_validation/check_ref_number_in_findings_text.py	`90%`	✅
audit/cross_validation/errors.py	`78%`	❌	`30` `77`
audit/cross_validation/naming.py	`93%`	✅	`197`
audit/cross_validation/submission_progress_check.py	`95%`	✅	`80`
audit/cross_validation/tribal_data_sharing_consent.py	`81%`	❌	`33` `36` `40`
audit/cross_validation/validate_general_information.py	`64%`	❌	`72` `76-79` `91` `94`
audit/fixtures/single_audit_checklist.py	`55%`	❌	`147-184` `230-239`
audit/intakelib/exceptions.py	`71%`	❌	`7-9` `12`
audit/intakelib/intermediate_representation.py	`91%`	✅	`27-28` `73` `91` `129` `200-203` `212-213` `283-284`
audit/intakelib/mapping_audit_findings.py	`97%`	✅	`55`
audit/intakelib/mapping_audit_findings_text.py	`97%`	✅	`54`
audit/intakelib/mapping_federal_awards.py	`93%`	✅	`92`
audit/intakelib/mapping_util.py	`79%`	❌	`21` `25` `29` `63` `99` `104-105` `114-120` `130` `145` `150`
audit/intakelib/checks/check_all_unique_award_numbers.py	`79%`	❌	`24`
audit/intakelib/checks/check_cluster_names.py	`75%`	❌	`20-25`
audit/intakelib/checks/check_cluster_total.py	`95%`	✅	`99`
audit/intakelib/checks/check_has_all_the_named_ranges.py	`95%`	✅	`66`
audit/intakelib/checks/check_is_a_workbook.py	`69%`	❌	`20`
audit/intakelib/checks/check_loan_balance_entries.py	`83%`	❌	`28`
audit/intakelib/checks/check_look_for_empty_rows.py	`91%`	✅	`18`
audit/intakelib/checks/check_no_major_program_no_type.py	`76%`	❌	`18` `27`
audit/intakelib/checks/check_no_repeat_findings.py	`76%`	❌	`21` `30`
audit/intakelib/checks/check_other_cluster_names.py	`81%`	❌	`23` `33`
audit/intakelib/checks/check_passthrough_name_when_no_direct.py	`88%`	✅	`9` `47`
audit/intakelib/checks/check_sequential_award_numbers.py	`82%`	❌	`25` `35`
audit/intakelib/checks/check_start_and_end_rows_of_all_columns_are_same.py	`89%`	✅	`14`
audit/intakelib/checks/check_state_cluster_names.py	`81%`	❌	`23` `33`
audit/intakelib/checks/check_version_number.py	`73%`	❌	`21` `31-32`
audit/intakelib/checks/runners.py	`97%`	✅	`155`
audit/intakelib/common/util.py	`90%`	✅	`22` `39`
audit/intakelib/transforms/xform_rename_additional_notes_sheet.py	`81%`	❌	`14`
audit/management/commands/load_fixtures.py	`46%`	❌	`39-45`
audit/models/models.py	`85%`	✅	`58` `60` `65` `67` `209` `215` `227` `239-242` `260` `437` `455-456` `464` `486` `584-585` `589` `597` `606` `612`
audit/views/audit_info_form_view.py	`27%`	❌	`25-74` `77-117` `120-137`
audit/views/manage_submission.py	`86%`	✅	`73-80`
audit/views/manage_submission_access.py	`98%`	✅	`113-114`
audit/views/pre_dissemination_download_view.py	`78%`	❌	`15-16` `21-22` `29-39`
audit/views/submission_progress_view.py	`89%`	✅	`111` `176-177`
audit/views/tribal_data_consent.py	`34%`	❌	`23-41` `44-79`
audit/views/unlock_after_certification.py	`57%`	❌	`28-51` `73-87`
audit/views/upload_report_view.py	`26%`	❌	`32-35` `44` `91-117` `120-170` `178-209`
audit/views/views.py	`53%`	❌	`74` `81-100` `123-124` `198-199` `220-230` `257` `268-269` `280-281` `283-287` `329-342` `345-359` `364-377` `394-400` `405-425` `452-456` `461-490` `533-537` `542-562` `589-593` `598-627` `670-674` `679-691` `694-704` `709-721` `754-768`
census_historical_migration/change_record.py	`96%`	✅	`30` `34`
census_historical_migration/test_federal_awards_xforms.py	`99%`	✅	`208-209`
census_historical_migration/sac_general_lib/audit_information.py	`91%`	✅	`24` `78`
census_historical_migration/sac_general_lib/cognizant_oversight.py	`68%`	❌	`11`
census_historical_migration/sac_general_lib/general_information.py	`91%`	✅	`150-151` `159-160` `204` `275`
census_historical_migration/sac_general_lib/sac_creator.py	`90%`	✅	`34`
census_historical_migration/sac_general_lib/utils.py	`83%`	❌	`33` `60-69`
census_historical_migration/transforms/xform_retrieve_uei.py	`67%`	❌	`10`
census_historical_migration/workbooklib/additional_eins.py	`84%`	❌	`58-60` `67-77`
census_historical_migration/workbooklib/additional_ueis.py	`77%`	❌	`27-29` `36-46`
census_historical_migration/workbooklib/corrective_action_plan.py	`81%`	❌	`40-42` `73-89`
census_historical_migration/workbooklib/excel_creation_utils.py	`65%`	❌	`99` `108-113` `118-125` `129-147` `160-164` `178-181`
census_historical_migration/workbooklib/federal_awards.py	`70%`	❌	`94-97` `268` `309-317` `327-352` `485-576`
census_historical_migration/workbooklib/findings.py	`27%`	❌	`40-45` `91-111` `116-136` `149-151` `158-179`
census_historical_migration/workbooklib/findings_text.py	`81%`	❌	`38-39` `70-87`
census_historical_migration/workbooklib/notes_to_sefa.py	`55%`	❌	`34-38` `101-102` `104-105` `107-108` `135-141` `149-157` `165-167` `172-189` `196-232`
config/error_handlers.py	`94%`	✅	`22`
config/test_settings.py	`92%`	✅	`33-34` `49-50`
config/urls.py	`72%`	❌	`88`
dissemination/file_downloads.py	`81%`	❌	`43-61` `91-93`
dissemination/forms.py	`87%`	✅	`135` `144` `255`
dissemination/search.py	`61%`	❌	`101` `108` `110` `113-121`
dissemination/summary_reports.py	`74%`	❌	`274` `300-302` `306-310` `421` `438` `459` `511-575` `601` `638-640` `663-670`
dissemination/test_search.py	`93%`	✅	`51-66` `473-474` `579-596` `608-632` `644-669` `677-693`
dissemination/test_summary_reports.py	`98%`	✅
dissemination/views.py	`76%`	❌	`134` `140-142` `159-223` `266` `296` `298` `332` `383` `385` `387` `465-470`
dissemination/migrations/0002_general_fac_accepted_date.py	`47%`	❌	`10-12`
dissemination/searchlib/search_alns.py	`37%`	❌	`44-58` `78-110` `115-177` `184-187`
dissemination/searchlib/search_direct_funding.py	`86%`	✅	`21-22`
dissemination/searchlib/search_findings.py	`76%`	❌	`18-24` `34` `36` `38`
dissemination/searchlib/search_general.py	`96%`	✅	`138`
dissemination/searchlib/search_passthrough_name.py	`35%`	❌	`21-31`
djangooidc/backends.py	`78%`	❌	`32` `57-63`
djangooidc/exceptions.py	`66%`	❌	`19` `21` `23` `28`
djangooidc/oidc.py	`16%`	❌	`32-35` `45-51` `64-70` `92-149` `153-199` `203-226` `230-275` `280-281` `286`
djangooidc/views.py	`81%`	❌	`22` `43` `109-110` `117`
djangooidc/tests/common.py	`97%`	✅
report_submission/forms.py	`92%`	✅	`35`
report_submission/views.py	`78%`	❌	`86` `225` `249-250` `255-256` `296-442` `445-455`
report_submission/templatetags/get_attr.py	`76%`	❌	`8` `11-14` `18`
support/admin.py	`88%`	✅	`76` `79` `84` `91-97` `100-102`
support/cog_over.py	`91%`	✅	`30-33` `93` `145`
support/test_admin_api.py	`81%`	❌	`23` `147-148` `237-238` `317-318`
support/test_cog_over.py	`98%`	✅	`134-135` `224`
support/management/commands/seed_cog_baseline.py	`98%`	✅	`20-21`
support/models/cog_over.py	`89%`	✅	`103-104`
tools/update_program_data.py	`89%`	✅	`96`
users/admin.py	`99%`	✅	`27`
users/auth.py	`96%`	✅	`58-59`
users/models.py	`96%`	✅	`18` `74-75`
users/fixtures/user_fixtures.py	`91%`	✅

Minimum allowed coverage is 85%

Generated by 🐒 cobertura-action against ebae523

github-actions · 2024-04-10T17:48:49Z

Terraform plan for meta

No changes. Your infrastructure matches the configuration.

No changes. Your infrastructure matches the configuration.

Terraform has compared your real infrastructure against your configuration
and found no differences, so no changes are needed.

Warning: Argument is deprecated

  with module.s3-backups.cloudfoundry_service_instance.bucket,
  on /tmp/terraform-data-dir/modules/s3-backups/s3/main.tf line 14, in resource "cloudfoundry_service_instance" "bucket":
  14:   recursive_delete = var.recursive_delete

Since CF API v3, recursive delete is always done on the cloudcontroller side.
This will be removed in future releases

✅ Plan applied in Deploy to Development and Management Environment #615

github-actions · 2024-04-10T17:48:53Z

Terraform plan for dev

No changes. Your infrastructure matches the configuration.

No changes. Your infrastructure matches the configuration.

Terraform has compared your real infrastructure against your configuration
and found no differences, so no changes are needed.

Warning: Argument is deprecated

  with module.dev.module.cg-logshipper.module.s3-logshipper-storage.cloudfoundry_service_instance.bucket,
  on /tmp/terraform-data-dir/modules/dev.cg-logshipper.s3-logshipper-storage/s3/main.tf line 14, in resource "cloudfoundry_service_instance" "bucket":
  14:   recursive_delete = var.recursive_delete

Since CF API v3, recursive delete is always done on the cloudcontroller side.
This will be removed in future releases

(and 3 more similar warnings elsewhere)

✅ Plan applied in Deploy to Development and Management Environment #615

phildominguez-gsa

Works well! Minor suggestions, take them or leave them.

backend/dissemination/forms.py

backend/dissemination/templates/search_filters/acceptance-date.html

Reformat search HTML, add four new filters

ae50c2d

jperson1 self-assigned this Apr 9, 2024

jperson1 had a problem deploying to dev April 9, 2024 20:43 — with GitHub Actions Failure

jperson1 had a problem deploying to meta April 9, 2024 20:43 — with GitHub Actions Failure

jperson1 added 2 commits April 10, 2024 13:46

Search - Tests for new filters

d8f0cb2

Search - Remove a comment, spacing

37d4ed2

jperson1 temporarily deployed to dev April 10, 2024 17:48 — with GitHub Actions Inactive

jperson1 temporarily deployed to meta April 10, 2024 17:48 — with GitHub Actions Inactive

jperson1 requested a review from a team April 10, 2024 17:57

jperson1 marked this pull request as ready for review April 10, 2024 17:57

jperson1 changed the title ~~[DRAFT] More Advanced Search Parameters & HTML Refactor~~ More Advanced Search Parameters & HTML Refactor Apr 10, 2024

Search filter - passthrough name

ece9ca7

jperson1 temporarily deployed to dev April 10, 2024 20:48 — with GitHub Actions Inactive

jperson1 temporarily deployed to meta April 10, 2024 20:48 — with GitHub Actions Inactive

jperson1 added 2 commits April 11, 2024 10:08

Search - fy_end_date to fy_end_month

8650b0e

Search - Type requirement hint and placeholder

d5220ca

jperson1 temporarily deployed to dev April 11, 2024 14:08 — with GitHub Actions Inactive

jperson1 temporarily deployed to meta April 11, 2024 14:08 — with GitHub Actions Inactive

Every good PR has a linting commit

8953207

jperson1 temporarily deployed to dev April 11, 2024 14:13 — with GitHub Actions Inactive

jperson1 temporarily deployed to meta April 11, 2024 14:13 — with GitHub Actions Inactive

Search tests - update fy_end_date for fy_end_month

493c894

jperson1 temporarily deployed to dev April 11, 2024 14:33 — with GitHub Actions Inactive

jperson1 temporarily deployed to meta April 11, 2024 14:33 — with GitHub Actions Inactive

jperson1 temporarily deployed to preview April 11, 2024 15:00 — with GitHub Actions Inactive

jperson1 temporarily deployed to preview April 11, 2024 15:03 — with GitHub Actions Inactive

phildominguez-gsa requested changes Apr 11, 2024

View reviewed changes

backend/dissemination/forms.py Outdated Show resolved Hide resolved

backend/dissemination/forms.py Outdated Show resolved Hide resolved

backend/dissemination/templates/search_filters/acceptance-date.html Outdated Show resolved Hide resolved

jperson1 added 3 commits April 11, 2024 12:28

Cog/over both option

3b039ea

Search filters - newlines on filter templates

9408485

Search - move text cleaning function to top level

57810dc

jperson1 temporarily deployed to dev April 11, 2024 16:43 — with GitHub Actions Inactive

jperson1 temporarily deployed to meta April 11, 2024 16:43 — with GitHub Actions Inactive

Search cog/over - "Both" -> "Either"

ebae523

jperson1 temporarily deployed to dev April 11, 2024 17:20 — with GitHub Actions Inactive

jperson1 temporarily deployed to meta April 11, 2024 17:20 — with GitHub Actions Inactive

phildominguez-gsa approved these changes Apr 11, 2024

View reviewed changes

jadudm added this pull request to the merge queue Apr 12, 2024

Merged via the queue into main with commit 06e3986 Apr 12, 2024
13 checks passed

jadudm deleted the jp/search-filter-refactor branch April 12, 2024 12:34

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

More Advanced Search Parameters & HTML Refactor #3623

More Advanced Search Parameters & HTML Refactor #3623

jperson1 commented Apr 9, 2024 •

edited

Loading

github-actions bot commented Apr 9, 2024 •

edited

Loading

github-actions bot commented Apr 10, 2024 •

edited

Loading

github-actions bot commented Apr 10, 2024 •

edited

Loading

phildominguez-gsa left a comment

More Advanced Search Parameters & HTML Refactor #3623

More Advanced Search Parameters & HTML Refactor #3623

Conversation

jperson1 commented Apr 9, 2024 • edited Loading

More Advanced Search Parameters & HTML Refactor

Changes:

How to test:

Screenshots:

PR checklist: submitters

PR checklist: reviewers

github-actions bot commented Apr 9, 2024 • edited Loading

github-actions bot commented Apr 10, 2024 • edited Loading

github-actions bot commented Apr 10, 2024 • edited Loading

phildominguez-gsa left a comment

Choose a reason for hiding this comment

jperson1 commented Apr 9, 2024 •

edited

Loading

github-actions bot commented Apr 9, 2024 •

edited

Loading

github-actions bot commented Apr 10, 2024 •

edited

Loading

github-actions bot commented Apr 10, 2024 •

edited

Loading