Restructure requirements for easier upstream syncing by SeanMeyer · Pull Request #222 · DataDog/lemur

SeanMeyer · 2026-01-29T16:08:27Z

Summary

This PR restructures the Python requirements files to use an overlay approach, making it easier to sync with upstream Netflix/lemur:

requirements-base.in: Direct copy from upstream Netflix/lemur requirements.in - can be overwritten with curl during upstream sync
requirements-datadog.in: Datadog-specific additions and security overrides - maintained separately
requirements.in: Simple include file that combines both

Datadog-specific additions include:

Internal packages (cert_orchestration_adapter)
Additional integrations (azure-*, google-cloud-compute, cert_manager, logmatic-python)
Version pins for compatibility (cryptography for FIPS, anyio, dnspython, etc.)
Security overrides not yet fixed in upstream

Security pins removed (upstream now satisfies):

requests (upstream has 2.32.5)
werkzeug (upstream has 3.1.5)
Flask-Cors (upstream has 6.0.2)
grpcio (upstream has 1.76.0)
sentry-sdk (upstream has 2.51.0)
pyjwt (upstream has 2.10.1)
zipp (upstream has 3.23.0)

Security pins remained (upstream still vulnerable):

jinja2 >= 3.1.6 (CVE-2024-56201, CVE-2024-56326, CVE-2025-27516) - upstream has 3.1.3
python_ldap >= 3.4.5 (CVE-2025-61911, CVE-2025-61912) - upstream has 3.4.4
h11 >= 0.16.0 (CVE-2025-43859)

Security pins added:

urllib3 >= 2.5.0 (CVE-2025-50181, CVE-2025-50182) - upstream has 1.26.18
This is the underlying purpose of the PR, adding this security fix.

Future Upstream Sync Workflow

# 1. Fetch latest upstream requirements
curl https://raw.githubusercontent.com/Netflix/lemur/main/requirements.in > requirements-base.in

# 2. Recompile all requirements
make up-reqs

# 3. Review changes and commit
git diff
git add -A && git commit -m "Sync requirements from upstream Netflix/lemur"

Test plan

Verify pip install -r requirements.txt works
Verify pip install -r requirements-tests.txt works
Run existing tests to confirm no regressions
Verify security pins are applied correctly (check resolved versions in requirements.txt)

SeanMeyer · 2026-01-29T16:08:33Z

@codex review

chatgpt-codex-connector · 2026-01-29T16:08:39Z

To use Codex here, create a Codex account and connect to github.

SeanMeyer · 2026-01-29T16:25:07Z

Local Verification Results

Check	Status
Security pins resolved correctly	✅ urllib3==2.6.3, jinja2==3.1.6, python-ldap==3.4.5, h11==0.16.0
`pip install -r requirements.txt`	✅ Installs successfully
`pip install -r requirements-tests.txt`	✅ Installs successfully
Unit tests	⚠️ Require PostgreSQL database (same behavior on master - CI will validate)

SeanMeyer · 2026-01-29T17:25:30Z

FYI I also have a smaller scope PR that just addresses the urllib3 vulns here: #221. But after thinking about it more I decided to go with this approach to sync from upstream. I'm open to whichever the maintainers of this repo prefer.

maperu

Validated the unit tests, lgtm

SeanMeyer · 2026-01-29T20:14:30Z

/merge

gh-worker-devflow-routing-ef8351 · 2026-01-29T20:14:38Z

View all feedbacks in Devflow UI.

2026-01-29 20:14:38 UTC ℹ️ Start processing command /merge

2026-01-29 20:15:06 UTC ℹ️ MergeQueue: waiting for PR to be ready

This pull request is not mergeable according to GitHub. Common reasons include pending required checks, missing approvals, or merge conflicts — but it could also be blocked by other repository rules or settings.
It will be added to the queue as soon as checks pass and/or get approvals. View in MergeQueue UI.
Note: if you pushed new commits since the last approval, you may need additional approval.
You can remove it from the waiting list with /remove command.

2026-01-29 20:28:39 UTC ⚠️ MergeQueue: This merge request was unqueued

maxime.perusse@datadoghq.com unqueued this merge request

maperu · 2026-01-29T20:28:10Z

/merge -c

gh-worker-devflow-routing-ef8351 · 2026-01-29T20:28:19Z

View all feedbacks in Devflow UI.

2026-01-29 20:28:19 UTC ℹ️ Start processing command /merge -c

maperu · 2026-01-29T20:52:35Z

/merge

gh-worker-devflow-routing-ef8351 · 2026-01-29T20:52:46Z

View all feedbacks in Devflow UI.

2026-01-29 20:52:45 UTC ℹ️ Start processing command /merge

2026-01-29 20:53:26 UTC ℹ️ MergeQueue: waiting for PR to be ready

This pull request is not mergeable according to GitHub. Common reasons include pending required checks, missing approvals, or merge conflicts — but it could also be blocked by other repository rules or settings.
It will be added to the queue as soon as checks pass and/or get approvals. View in MergeQueue UI.
Note: if you pushed new commits since the last approval, you may need additional approval.
You can remove it from the waiting list with /remove command.

2026-01-30 00:54:21 UTC ⚠️ MergeQueue: This merge request was unqueued

devflow unqueued this merge request: It did not become mergeable within the expected time

maperu · 2026-01-29T21:02:59Z

/merge

gh-worker-devflow-routing-ef8351 · 2026-01-29T21:03:07Z

View all feedbacks in Devflow UI.

2026-01-29 21:03:06 UTC ℹ️ Start processing command /merge

2026-01-29 21:03:10 UTC ❌ MergeQueue

PR already in the queue with status waiting

maperu · 2026-02-02T15:25:39Z

/merge

gh-worker-devflow-routing-ef8351 · 2026-02-02T15:25:44Z

View all feedbacks in Devflow UI.

2026-02-02 15:25:44 UTC ℹ️ Start processing command /merge

2026-02-02 15:25:51 UTC ℹ️ MergeQueue: waiting for PR to be ready

This pull request is not mergeable according to GitHub. Common reasons include pending required checks, missing approvals, or merge conflicts — but it could also be blocked by other repository rules or settings.
It will be added to the queue as soon as checks pass and/or get approvals. View in MergeQueue UI.
Note: if you pushed new commits since the last approval, you may need additional approval.
You can remove it from the waiting list with /remove command.

2026-02-02 15:27:19 UTC ⚠️ MergeQueue: This merge request was unqueued

maxime.perusse@datadoghq.com unqueued this merge request

maperu · 2026-02-02T15:27:09Z

/merge -c

gh-worker-devflow-routing-ef8351 · 2026-02-02T18:49:34Z

View all feedbacks in Devflow UI.

2026-02-02 18:49:34 UTC ℹ️ Start processing command /remove

2026-02-02 18:49:37 UTC ℹ️ Devflow: /remove

maperu · 2026-02-02T18:58:39Z

/merge

gh-worker-devflow-routing-ef8351 · 2026-02-02T18:58:43Z

View all feedbacks in Devflow UI.

2026-02-02 18:58:42 UTC ℹ️ Start processing command /merge

2026-02-02 18:58:49 UTC ℹ️ MergeQueue: waiting for PR to be ready

This pull request is not mergeable according to GitHub. Common reasons include pending required checks, missing approvals, or merge conflicts — but it could also be blocked by other repository rules or settings.
It will be added to the queue as soon as checks pass and/or get approvals. View in MergeQueue UI.
Note: if you pushed new commits since the last approval, you may need additional approval.
You can remove it from the waiting list with /remove command.

2026-02-02 19:37:38 UTC ℹ️ MergeQueue: merge request added to the queue

The expected merge time in master is approximately 0s (p90).

2026-02-02 19:53:24 UTC ℹ️ MergeQueue: This merge request was merged

maperu · 2026-02-02T19:22:27Z

/merge

gh-worker-devflow-routing-ef8351 · 2026-02-02T19:22:31Z

View all feedbacks in Devflow UI.

2026-02-02 19:22:31 UTC ℹ️ Start processing command /merge

2026-02-02 19:22:34 UTC ❌ MergeQueue

PR already in the queue with status waiting

Split requirements.in into: - requirements-base.in: synced from upstream Netflix/lemur - requirements-datadog.in: Datadog-specific additions and security overrides This makes future upstream syncs easier - just overwrite requirements-base.in from upstream and recompile, no merge conflicts. Removed redundant security pins that upstream now satisfies: - requests (upstream has 2.32.5) - werkzeug (upstream has 3.1.5) - Flask-Cors (upstream has 6.0.2) - grpcio (upstream has 1.76.0) - sentry-sdk (upstream has 2.51.0) - pyjwt (upstream has 2.10.1) - zipp (upstream has 3.23.0) Kept security pins still needed (upstream vulnerable): - urllib3 >= 2.5.0 (upstream has 1.26.18) - jinja2 >= 3.1.6 (upstream has 3.1.3) - python_ldap >= 3.4.5 (upstream has 3.4.4) - h11 >= 0.16.0 Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

- Add SQLALCHEMY_ENGINE_OPTIONS to test config with connection/statement timeouts to prevent test fixture cleanup from hanging indefinitely - Fix exception handling in verify.py: "except ConnectionError or Timeout" only catches ConnectionError in Python. Changed to "except (ConnectionError, Timeout)"

- Convert user ID to string when creating JWT payload to ensure consistency. - Update user retrieval logic to handle string user IDs by converting them back to integers before fetching from the user service. - Adjust test vectors to reflect updated JWT tokens for valid users and admins.

josepy.util module is now a deprecation wrapper in newer versions. Patch the class where it's actually imported in acme_handlers.py instead.

The request_certificate method doesn't use ComparableX509 (only revoke_certificate does). Removing the mock fixes compatibility with newer josepy versions.

PyJWT 2.10+ requires 'sub' to be a string, but existing tokens have integer sub. Adding verify_sub=False allows both old (int) and new (string) tokens to work. The isinstance check in login_required handles normalization for user lookup.

SeanMeyer marked this pull request as ready for review January 29, 2026 17:23

SeanMeyer requested a review from a team as a code owner January 29, 2026 17:23

maperu previously approved these changes Jan 29, 2026

View reviewed changes

dd-devflow bot added the mergequeue-status: waiting label Jan 29, 2026

dd-devflow bot added mergequeue-status: removed and removed mergequeue-status: waiting labels Jan 29, 2026

dd-devflow bot added mergequeue-status: waiting and removed mergequeue-status: removed labels Jan 29, 2026

maperu dismissed their stale review via 74e26cc January 29, 2026 20:57

maperu previously approved these changes Jan 29, 2026

View reviewed changes

dd-devflow bot added mergequeue-status: removed and removed mergequeue-status: waiting labels Jan 30, 2026

maperu dismissed their stale review via eea2cfa January 30, 2026 21:33

dd-devflow bot added mergequeue-status: waiting and removed mergequeue-status: removed labels Feb 2, 2026

dd-devflow bot added mergequeue-status: removed and removed mergequeue-status: waiting labels Feb 2, 2026

maperu disabled auto-merge February 2, 2026 18:49

dd-devflow bot added mergequeue-status: waiting and removed mergequeue-status: removed labels Feb 2, 2026

SeanMeyer and others added 12 commits February 2, 2026 14:25

Trigger CI pipeline

2908c58

Trigger CI pipeline

f39fc16

Fix flake8 F824: remove unused global statement

770bea6

Trigger CI pipeline

73b3637

Fix ACME test: patch ComparableX509 at correct import location

81b1226

josepy.util module is now a deprecation wrapper in newer versions. Patch the class where it's actually imported in acme_handlers.py instead.

Trigger CI pipeline

7fe5aea

Remove unnecessary ComparableX509 mock from test_request_certificate

e58ce54

The request_certificate method doesn't use ComparableX509 (only revoke_certificate does). Removing the mock fixes compatibility with newer josepy versions.

Trigger CI pipeline

83d6b83

maperu force-pushed the restructure-requirements-overlay branch from 4ccceaf to 8d49cae Compare February 2, 2026 19:25

dd-devflow bot added mergequeue-status: queued mergequeue-status: in_progress and removed mergequeue-status: waiting mergequeue-status: queued labels Feb 2, 2026

gh-worker-dd-mergequeue-cf854d bot merged commit 107ba25 into DataDog:master Feb 2, 2026
10 checks passed

dd-devflow bot added mergequeue-status: done and removed mergequeue-status: in_progress labels Feb 2, 2026

Conversation

SeanMeyer commented Jan 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Datadog-specific additions include:

Security pins removed (upstream now satisfies):

Security pins remained (upstream still vulnerable):

Security pins added:

Future Upstream Sync Workflow

Test plan

Uh oh!

SeanMeyer commented Jan 29, 2026

Uh oh!

chatgpt-codex-connector bot commented Jan 29, 2026

Uh oh!

SeanMeyer commented Jan 29, 2026

Local Verification Results

Uh oh!

SeanMeyer commented Jan 29, 2026

Uh oh!

maperu left a comment

Choose a reason for hiding this comment

Uh oh!

SeanMeyer commented Jan 29, 2026

Uh oh!

gh-worker-devflow-routing-ef8351 bot commented Jan 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

maperu commented Jan 29, 2026

Uh oh!

gh-worker-devflow-routing-ef8351 bot commented Jan 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

maperu commented Jan 29, 2026

Uh oh!

gh-worker-devflow-routing-ef8351 bot commented Jan 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

maperu commented Jan 29, 2026

Uh oh!

gh-worker-devflow-routing-ef8351 bot commented Jan 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

maperu commented Feb 2, 2026

Uh oh!

gh-worker-devflow-routing-ef8351 bot commented Feb 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

maperu commented Feb 2, 2026

Uh oh!

gh-worker-devflow-routing-ef8351 bot commented Feb 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

maperu commented Feb 2, 2026

Uh oh!

gh-worker-devflow-routing-ef8351 bot commented Feb 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

maperu commented Feb 2, 2026

Uh oh!

gh-worker-devflow-routing-ef8351 bot commented Feb 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

SeanMeyer commented Jan 29, 2026 •

edited

Loading

gh-worker-devflow-routing-ef8351 bot commented Jan 29, 2026 •

edited

Loading

gh-worker-devflow-routing-ef8351 bot commented Jan 29, 2026 •

edited

Loading

gh-worker-devflow-routing-ef8351 bot commented Jan 29, 2026 •

edited

Loading

gh-worker-devflow-routing-ef8351 bot commented Jan 29, 2026 •

edited

Loading

gh-worker-devflow-routing-ef8351 bot commented Feb 2, 2026 •

edited

Loading

gh-worker-devflow-routing-ef8351 bot commented Feb 2, 2026 •

edited

Loading

gh-worker-devflow-routing-ef8351 bot commented Feb 2, 2026 •

edited

Loading

gh-worker-devflow-routing-ef8351 bot commented Feb 2, 2026 •

edited

Loading