Skip to content

Eng 319 verify update all logging for dsr activity logging#6191

Merged
JadeCara merged 11 commits intomainfrom
ENG-319-verify-update-all-logging-for-dsr-activity-logging
Jun 3, 2025
Merged

Eng 319 verify update all logging for dsr activity logging#6191
JadeCara merged 11 commits intomainfrom
ENG-319-verify-update-all-logging-for-dsr-activity-logging

Conversation

@JadeCara
Copy link
Copy Markdown
Contributor

@JadeCara JadeCara commented Jun 3, 2025

Issue 319

Description Of Changes

Adds Execution Logs for uploading the access packages as well as some bind logs.
Additional changes for this ticket in fidesplus PR 2211

Code Changes

  • Added Execution logs for uploading access packages.
  • Added bind logs for uploading access packages.
  • Added a check for the privacy request status to ensure it wont try to complete while in error status.
  • Updated the upload functions to raise a StorageUploadError and include the specific error messages

Steps to Confirm

  1. Complete an access DSR and verify that the logging appears on the Privacy request page:
    a. with an error - Upload error appears in the UI and in logging. To create this error I went and changed the last two digits of the storage config default using the /api/v1/storage/default PUT method
    Screenshot 2025-06-03 at 11 26 24 AM
fidesplus-slim  | 2025-06-03 17:26:05.631 | ERROR    | 
fides.api.service.privacy_request.request_runner_service:upload_access_results:275 - 
Error uploading subject access data for rule {rule_key} on policy {policy_key}: {error} | 
{'privacy_request_id': 'pri_070dee05-70e0-44c6-a74a-f8450policy_key}: {error} | 
{
    'privacy_request_id': 'pri_070dee05-70e0-44c6-a74a-f8450aa0a963', 
    'privacy_request_source': 'Request Manager', 
    'rule_key': 'default_access_policy_rule', 
    'policy_key': 'default_access_policy', 
    'error': 'Error uploading to Google Cloud Storage: 404 POST ...'}

b. with success - Upload appears in the UI and in logging
Screenshot 2025-06-03 at 11 12 29 AM

fidesplus-slim  | 2025-06-03 17:12:07.212 | INFO     | 
fides.api.service.privacy_request.request_runner_service:upload_access_results:267 - 
Access Package Upload successful for privacy request: pri_82dea3f7-257b-4d3c-b454-3a0f719f58e9. 
Time taken: 0.48978304862976074 seconds | 
{
    'privacy_request_id': 'pri_82dea3f7-257b-4d3c-b454-3a0f719f58e9', 
    'privacy_request_source': 'Request Manager', 
    'time_taken': 0.48976874351501465
}

Pre-Merge Checklist

  • Issue requirements met
  • All CI pipelines succeeded
  • CHANGELOG.md updated
    • Add a db-migration This indicates that a change includes a database migration label to the entry if your change includes a DB migration
    • Add a high-risk This issue suggests changes that have a high-probability of breaking existing code label to the entry if your change includes a high-risk change (i.e. potential for performance impact or unexpected regression) that should be flagged
    • Updates unreleased work already in Changelog, no new entry necessary
  • Followup issues:
    • Followup issues created
    • No followup issues
  • Database migrations:
    • Ensure that your downrev is up to date with the latest revision on main
    • Ensure that your downgrade() migration is correct and works
      • If a downgrade migration is not possible for this change, please call this out in the PR description!
    • No migrations
  • Documentation:
    • Documentation complete, PR opened in fidesdocs
    • Documentation issue created in fidesdocs
    • If there are any new client scopes created as part of the pull request, remember to update public-facing documentation that references our scope registry
    • No documentation updates required

Jade Wibbels added 2 commits May 30, 2025 10:18
@vercel
Copy link
Copy Markdown
Contributor

vercel Bot commented Jun 3, 2025

The latest updates on your projects. Learn more about Vercel for Git ↗︎

2 Skipped Deployments
Name Status Preview Comments Updated (UTC)
fides-plus-nightly ⬜️ Ignored (Inspect) Visit Preview Jun 3, 2025 8:48pm
fides-privacy-center ⬜️ Ignored (Inspect) Jun 3, 2025 8:48pm

@codecov
Copy link
Copy Markdown

codecov Bot commented Jun 3, 2025

Codecov Report

Attention: Patch coverage is 72.22222% with 5 lines in your changes missing coverage. Please review.

Project coverage is 87.07%. Comparing base (3509738) to head (41d409a).
Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
.../service/privacy_request/request_runner_service.py 78.57% 2 Missing and 1 partial ⚠️
src/fides/api/tasks/storage.py 50.00% 2 Missing ⚠️

❌ Your patch status has failed because the patch coverage (72.22%) is below the target coverage (100.00%). You can increase the patch coverage or adjust the target coverage.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #6191      +/-   ##
==========================================
- Coverage   87.08%   87.07%   -0.01%     
==========================================
  Files         423      423              
  Lines       26315    26321       +6     
  Branches     2869     2870       +1     
==========================================
+ Hits        22916    22919       +3     
- Misses       2775     2776       +1     
- Partials      624      626       +2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@JadeCara JadeCara marked this pull request as ready for review June 3, 2025 18:50
Copy link
Copy Markdown
Contributor

@erosselli erosselli left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving with some questions. Don't forget the changelog entry

Comment on lines -14 to -16
attachment_uploaded = "attachment_uploaded"
attachment_deleted = "attachment_deleted"
attachment_retrieved = "attachment_retrieved"
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

are these not used anymore?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added them and then learned more about the audit log so they were never really used.

Comment on lines +264 to +266
logger.bind(
privacy_request_id=privacy_request.id,
time_taken=time.time() - start_time,
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think there might be a decorator you can use to bind the privacy request id all throughout the function?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated with log_context decorator! Thanks :)

connection_key=None,
dataset_name="Access Package Upload",
collection_name=None,
message=f"Access Package Upload successful for privacy request: {privacy_request.id}",
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we need the id for the privacy request in the message, if the execution log is already linked to the privacy request? thinking about the UX of reading these execution logs in the UI

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated - much cleaner now :)

Comment on lines +625 to +626
# Only mark as complete if not in error state
if privacy_request.status != PrivacyRequestStatus.error:
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm surprised we didn't have this check before ?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was too! I purposefully made the upload error for the test steps above it errored and then completed.

@JadeCara JadeCara merged commit 0bfda78 into main Jun 3, 2025
39 of 40 checks passed
@JadeCara JadeCara deleted the ENG-319-verify-update-all-logging-for-dsr-activity-logging branch June 3, 2025 21:22
@cypress
Copy link
Copy Markdown

cypress Bot commented Jun 3, 2025

fides    Run #12957

Run Properties:  status check passed Passed #12957  •  git commit 0bfda78b61: Eng 319 verify update all logging for dsr activity logging (#6191)
Project fides
Branch Review main
Run status status check passed Passed #12957
Run duration 01m 00s
Commit git commit 0bfda78b61: Eng 319 verify update all logging for dsr activity logging (#6191)
Committer JadeWibbels
View all properties for this run ↗︎

Test results
Tests that failed  Failures 0
Tests that were flaky  Flaky 0
Tests that did not run due to a developer annotating a test with .skip  Pending 0
Tests that did not run due to a failure in a mocha hook  Skipped 0
Tests that passed  Passing 5
View all changes introduced in this branch ↗︎

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants