Update Python dependency versions identified by dependabot#1481
Update Python dependency versions identified by dependabot#1481landonshumway-ia wants to merge 6 commits intocsg-org:mainfrom
Conversation
Introduces major updates: Marshmallow 3.x -> 4.x
|
Warning Rate limit exceeded
Your organization is not enrolled in usage-based pricing. Contact your admin to enable usage-based pricing to continue reviews beyond the rate limit, or try again in 10 minutes and 43 seconds. ⌛ How to resolve this issue?After the wait time has elapsed, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. 🚦 How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout. Please see our FAQ for further information. ℹ️ Review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (4)
📝 WalkthroughWalkthroughThis pull request updates multiple backend project dependencies including AWS SDKs, Lambda Powertools, and Marshmallow, while converting datetime schema fields across both compact-connect and cosmetology-app projects from Marshmallow's generic Changes
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes The changes span multiple files with a mix of repetitive dependency version bumps (low cognitive load) and systematic datetime field type conversions across many schemas (moderate cognitive load). While the datetime conversions follow a consistent pattern, each schema file requires verification that all appropriate datetime fields are updated consistently. The Marshmallow 4.x upgrade from 3.26.2 requires particular attention for compatibility. Test fixture updates need validation against the new datetime format expectations. Possibly related PRs
Suggested reviewers
Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Now that we are moving from Marshmallow 3.x -> 4.x, the datetime fields are more permissive, and will convert a bare date string YYYY-MM-DD into a naive datetime format, which we do not want. We need to enforce that these fields are timezone aware to avoid any future bugs/issues inserting naive timestamps.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
backend/compact-connect/lambdas/python/common/cc_common/data_model/schema/privilege/record.py (1)
135-142:⚠️ Potential issue | 🟠 Major
_enforce_datetimesno longer provides the backwards-compat guarantee it claims under Marshmallow 4.
ensure_value_is_datetime(incommon.py) only upgrades length-10 date strings (YYYY-MM-DD) to UTC-aware datetime strings — if the input is already a datetime-shaped string, it returns it unchanged. That was fine when these fields wereDateTime(tolerant of naive ISO strings), but now withAwareDateTimeany historical record whosedateOfIssuance/dateOfRenewalwas written as a naive datetime string (e.g.2024-01-02T03:04:05) will bypass this hook's "fix up" and then failAwareDateTimevalidation on load.Given this is the schema that most explicitly advertises legacy-data tolerance, please either:
- Extend
ensure_value_is_datetimeto also coerce naive datetime strings to UTC-aware (parse, settzinfo=UTCif missing, re-serialize), or- Confirm via a migration/backfill that no naive datetime values remain for these fields in production.
Option 1 is minimally invasive and preserves the safety net already described in the docstring. It also transparently fixes the same risk in other schemas that call into
ensure_value_is_datetime.♻️ Sketch of the extended helper
def ensure_value_is_datetime(value: str): ... dt = datetime.fromisoformat(value) if len(value) == 10: value_dt = datetime.combine(dt, datetime.max.time(), tzinfo=UTC).replace(microsecond=0) return value_dt.isoformat() - # Not a date string, return the original - return value + # Promote naive datetime strings to UTC-aware for Marshmallow 4 AwareDateTime compatibility + if dt.tzinfo is None: + return dt.replace(tzinfo=UTC).isoformat() + return valueAlso note the updated comment on line 138 now reads "for dateOfRenewal and dateOfIssuance to datetime values" but the real intent after this PR is "to timezone-aware datetime values" — worth tightening.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@backend/compact-connect/lambdas/python/common/cc_common/data_model/schema/privilege/record.py` around lines 135 - 142, The _enforce_datetimes backward-compat guarantee breaks under Marshmallow 4 because ensure_value_is_datetime currently only upgrades YYYY-MM-DD strings and leaves naive ISO datetime strings unchanged, causing AwareDateTime validation failures; update ensure_value_is_datetime (used by _enforce_datetimes) to parse ISO datetime strings and, if they lack timezone info, attach UTC (e.g., parse -> set tzinfo=UTC -> re-serialize/return timezone-aware datetime) so dateOfIssuance and dateOfRenewal are always timezone-aware before AwareDateTime validation, and update the doc/comment in _enforce_datetimes to say "timezone-aware datetime values"; ensure function names referenced: ensure_value_is_datetime, _enforce_datetimes, and fields dateOfIssuance/dateOfRenewal/AwareDateTime.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In
`@backend/cosmetology-app/lambdas/python/common/cc_common/data_model/schema/license/ingest.py`:
- Line 3: The AwareDateTime field used for eventTime in
SanitizedLicenseIngestDataEventSchema may reject timezone-naive timestamps;
update the schema's eventTime AwareDateTime declaration to set
default_timezone=timezone.utc so Marshmallow will treat naive datetimes as UTC
when loading (import timezone from datetime as needed) and ensure existing
producers' offset-bearing strings continue to validate unchanged.
In `@backend/cosmetology-app/lambdas/python/common/requirements.in`:
- Line 5: Replace the existing dependency line "marshmallow>=3.21.3, <5.0.0"
with a constraint that excludes the vulnerable 4.0.0–4.1.1 releases; either use
the two-platform-specific lines "marshmallow>=3.21.3, <4.0.0; python_version <
'3.x'" and "marshmallow>=4.1.2, <5.0.0; python_version >= '3.x'" or a single
unified exclusion "marshmallow>=3.21.3, <5.0.0, !=4.0.0, !=4.0.1, !=4.1.0,
!=4.1.1" in place of the original "marshmallow>=3.21.3, <5.0.0" entry so the
vulnerable CVE-2025-68480 versions are not installable.
---
Outside diff comments:
In
`@backend/compact-connect/lambdas/python/common/cc_common/data_model/schema/privilege/record.py`:
- Around line 135-142: The _enforce_datetimes backward-compat guarantee breaks
under Marshmallow 4 because ensure_value_is_datetime currently only upgrades
YYYY-MM-DD strings and leaves naive ISO datetime strings unchanged, causing
AwareDateTime validation failures; update ensure_value_is_datetime (used by
_enforce_datetimes) to parse ISO datetime strings and, if they lack timezone
info, attach UTC (e.g., parse -> set tzinfo=UTC -> re-serialize/return
timezone-aware datetime) so dateOfIssuance and dateOfRenewal are always
timezone-aware before AwareDateTime validation, and update the doc/comment in
_enforce_datetimes to say "timezone-aware datetime values"; ensure function
names referenced: ensure_value_is_datetime, _enforce_datetimes, and fields
dateOfIssuance/dateOfRenewal/AwareDateTime.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 4f61e516-79f0-49f5-8094-691d74958777
📒 Files selected for processing (48)
backend/compact-connect/lambdas/python/cognito-backup/requirements-dev.txtbackend/compact-connect/lambdas/python/common/cc_common/data_model/schema/adverse_action/record.pybackend/compact-connect/lambdas/python/common/cc_common/data_model/schema/base_record.pybackend/compact-connect/lambdas/python/common/cc_common/data_model/schema/data_event/api.pybackend/compact-connect/lambdas/python/common/cc_common/data_model/schema/investigation/record.pybackend/compact-connect/lambdas/python/common/cc_common/data_model/schema/license/ingest.pybackend/compact-connect/lambdas/python/common/cc_common/data_model/schema/license/record.pybackend/compact-connect/lambdas/python/common/cc_common/data_model/schema/military_affiliation/record.pybackend/compact-connect/lambdas/python/common/cc_common/data_model/schema/privilege/record.pybackend/compact-connect/lambdas/python/common/cc_common/data_model/schema/provider/api.pybackend/compact-connect/lambdas/python/common/cc_common/data_model/schema/provider/record.pybackend/compact-connect/lambdas/python/common/requirements-dev.txtbackend/compact-connect/lambdas/python/common/requirements.inbackend/compact-connect/lambdas/python/common/requirements.txtbackend/compact-connect/lambdas/python/compact-configuration/requirements-dev.txtbackend/compact-connect/lambdas/python/custom-resources/requirements-dev.txtbackend/compact-connect/lambdas/python/data-events/requirements-dev.txtbackend/compact-connect/lambdas/python/disaster-recovery/requirements-dev.txtbackend/compact-connect/lambdas/python/provider-data-v1/requirements-dev.txtbackend/compact-connect/lambdas/python/search/requirements-dev.txtbackend/compact-connect/lambdas/python/search/tests/function/test_expiration_reminders.pybackend/compact-connect/lambdas/python/staff-user-pre-token/requirements-dev.txtbackend/compact-connect/lambdas/python/staff-users/requirements-dev.txtbackend/compact-connect/requirements-dev.txtbackend/compact-connect/requirements.txtbackend/cosmetology-app/lambdas/python/cognito-backup/requirements-dev.txtbackend/cosmetology-app/lambdas/python/cognito-backup/requirements.txtbackend/cosmetology-app/lambdas/python/common/cc_common/data_model/schema/adverse_action/record.pybackend/cosmetology-app/lambdas/python/common/cc_common/data_model/schema/base_record.pybackend/cosmetology-app/lambdas/python/common/cc_common/data_model/schema/data_event/api.pybackend/cosmetology-app/lambdas/python/common/cc_common/data_model/schema/investigation/record.pybackend/cosmetology-app/lambdas/python/common/cc_common/data_model/schema/license/ingest.pybackend/cosmetology-app/lambdas/python/common/cc_common/data_model/schema/license/record.pybackend/cosmetology-app/lambdas/python/common/cc_common/data_model/schema/provider/record.pybackend/cosmetology-app/lambdas/python/common/requirements-dev.inbackend/cosmetology-app/lambdas/python/common/requirements-dev.txtbackend/cosmetology-app/lambdas/python/common/requirements.inbackend/cosmetology-app/lambdas/python/common/requirements.txtbackend/cosmetology-app/lambdas/python/compact-configuration/requirements-dev.txtbackend/cosmetology-app/lambdas/python/custom-resources/requirements-dev.txtbackend/cosmetology-app/lambdas/python/data-events/requirements-dev.txtbackend/cosmetology-app/lambdas/python/disaster-recovery/requirements-dev.txtbackend/cosmetology-app/lambdas/python/provider-data-v1/requirements-dev.txtbackend/cosmetology-app/lambdas/python/search/requirements-dev.txtbackend/cosmetology-app/lambdas/python/staff-user-pre-token/requirements-dev.txtbackend/cosmetology-app/lambdas/python/staff-users/requirements-dev.txtbackend/cosmetology-app/requirements-dev.txtbackend/cosmetology-app/requirements.txt
💤 Files with no reviewable changes (1)
- backend/cosmetology-app/lambdas/python/common/requirements-dev.in
| # ruff: noqa: N801, N815, ARG002 invalid-name unused-argument | ||
| from marshmallow import ValidationError, pre_load, validates_schema | ||
| from marshmallow.fields import UUID, Date, DateTime, String | ||
| from marshmallow.fields import UUID, AwareDateTime, Date, String |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# Find producers that publish license ingest events
rg -nP --type=py '\beventTime\b' -C3 -g '!**/tests/**'Repository: csg-org/CompactConnect
Length of output: 50378
🏁 Script executed:
# First, let's examine the actual schema file mentioned in the review
cat -n backend/cosmetology-app/lambdas/python/common/cc_common/data_model/schema/license/ingest.py | head -100Repository: csg-org/CompactConnect
Length of output: 4257
🏁 Script executed:
# Check what config.current_standard_datetime is
rg -nP 'current_standard_datetime' backend/cosmetology-app -A2 -B2 --type=py | head -80Repository: csg-org/CompactConnect
Length of output: 8989
🏁 Script executed:
# Look for config definition and current_standard_datetime
fd -type f -name 'config.py' | xargs grep -l 'current_standard_datetime' 2>/dev/nullRepository: csg-org/CompactConnect
Length of output: 238
🏁 Script executed:
# Find config files that might define current_standard_datetime
find . -type f -name '*.py' -path '*/cc_common/*' | xargs grep -l 'def current_standard_datetime\|current_standard_datetime =' 2>/dev/null | head -5Repository: csg-org/CompactConnect
Length of output: 201
🏁 Script executed:
# Search for config class definition
rg -nP 'class.*Config|current_standard_datetime.*=' backend/cosmetology-app/lambdas/python/common/cc_common -B3 -A3 --type=py | head -100Repository: csg-org/CompactConnect
Length of output: 8633
🏁 Script executed:
# Let's look at the actual config module
find . -path '*/cc_common/config.py' | xargs cat -n 2>/dev/nullRepository: csg-org/CompactConnect
Length of output: 26358
AwareDateTime on eventTime — current producers are timezone-aware; consider adding default_timezone=timezone.utc for backward compatibility with potential in-flight messages.
SanitizedLicenseIngestDataEventSchema loads from SQS using Marshmallow 4.x, where AwareDateTime rejects timezone-naive input. Verification shows all identified producers already emit offset-bearing ISO strings via config.current_standard_datetime.isoformat() (e.g., 2024-11-08T23:59:59+00:00), so there is no immediate validation failure risk. However, to safely handle any in-flight messages from earlier rollouts that may lack timezone info, consider adding default_timezone=timezone.utc to this field to provide graceful fallback during transition.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In
`@backend/cosmetology-app/lambdas/python/common/cc_common/data_model/schema/license/ingest.py`
at line 3, The AwareDateTime field used for eventTime in
SanitizedLicenseIngestDataEventSchema may reject timezone-naive timestamps;
update the schema's eventTime AwareDateTime declaration to set
default_timezone=timezone.utc so Marshmallow will treat naive datetimes as UTC
when loading (import timezone from datetime as needed) and ensure existing
producers' offset-bearing strings continue to validate unchanged.
1 participant
In an effort to close out the list of Dependabot generated PRs, this introduces python dependency updates that dependabot has identified.
This includes a list of minor updates, as well as the following major updates:
Marshmallow 3.x -> 4.x see changelog for description of breaking changes
The biggest change for this was replacing the DateTime marshmallow field type with the AwareDateTime type. The DateTime marshmallow type, as of 4.x, is much more permissive and will allow date strings (YYYY-MM-DD) to silently pass validation and will set it as a naive datetime string, which we want to avoid as our datetimes need to be timezone aware to support converting to various local timezones. Scanning through our identified writes to the current datetime fields showed that we already enforce timezone awareness when storing datetime fields, and we migrated off of naive timestamps before we even launched in prod, so it is expected that this is a backwards compatible change with our production environment. Even so, extensive smoke testing, UI testing, and zap scanning should be performed against this change before releasing to prod (unfortunately, given that this is a core schema related update, the feature flag framework will not be feasible to use at this fundamental scope without impacting system performance).
We were not able to update the attrs dependency to the latest 26.x version, as this is a transitive dependency of jsii (used by CDK). Until jsii ships a new version that accepts attrs 26, the CDK app cannot move to attrs 26, so the shared venv can't either. For now it will remain as is and we will have Dependabot ignore that update. The only other location we use it is as a dev dependency for moto in the common tests, so it is not bundled in the runtime code.
Closes #1319