Fixes #21941: Use awsSessionToken as plain str in OpenSearch connector#27465
Fixes #21941: Use awsSessionToken as plain str in OpenSearch connector#27465Megh-Shah-08 wants to merge 3 commits intoopen-metadata:mainfrom
Conversation
…get_secret_value() (open-metadata#21941)
|
Hi there 👋 Thanks for your contribution! The OpenMetadata team will review the PR shortly! Once it has been labeled as Let us know if you need any help! |
There was a problem hiding this comment.
Pull request overview
Fixes ingestion failures for the OpenSearch connector when using AWS temporary credentials by treating awsSessionToken as a plain string (per the AWSCredentials schema), avoiding an invalid .get_secret_value() call.
Changes:
- Update OpenSearch AWS IAM auth to pass
awsSessionTokendirectly asstr. - Minor formatting adjustment in Basic Auth tuple construction (no behavioral change).
|
Hi there 👋 Thanks for your contribution! The OpenMetadata team will review the PR shortly! Once it has been labeled as Let us know if you need any help! |
ulixius9
added
the
safe to test
|
The Python checkstyle failed. Please run You can install the pre-commit hooks with |
🔴 Playwright Results — 1 failure(s), 22 flaky✅ 3642 passed · ❌ 1 failed · 🟡 22 flaky · ⏭️ 84 skipped
Genuine Failures (failed on all attempts)❌
|
![gitar-bot[bot]](https://avatars.githubusercontent.com/in/827041?s=60&v=4){kind=small}
|
|
||
| [tool.black] | ||
| extend-exclude = "src/metadata/generated" | ||
| exclude = "src/metadata/generated|env|venv" |
There was a problem hiding this comment.
⚠️ Bug: Switching from extend-exclude to exclude drops Black/pycln defaults
Changing extend-exclude to exclude in both [tool.black] and [tool.pycln] causes these tools to lose their built-in default exclusions (.git, .tox, .mypy_cache, .pytest_cache, __pypackages__, build, dist, etc.). Only src/metadata/generated, env, and venv will be excluded.
The intent was to add env/venv to the exclusion list, but extend-exclude is the correct directive for that — it adds patterns on top of the defaults. Use extend-exclude and append the new patterns.
Suggested fix:
[tool.black]
extend-exclude = "src/metadata/generated|env|venv"
[tool.pycln]
all = true
extend-exclude = "src/metadata/generated|env|venv"
Was this helpful? React with 👍 / 👎 | Reply gitar fix to apply this suggestion
|
|
||
| [tool.black] | ||
| extend-exclude = "src/metadata/generated" | ||
| exclude = "src/metadata/generated|env|venv" |
There was a problem hiding this comment.
[tool.black] exclude = ... replaces Black’s default exclude regex (which normally skips .nox/.tox/.git/build/dist/etc.). Since nox runs black --check ., this will make lint traverse virtualenv and build artifacts and can drastically slow down or fail CI. Use extend-exclude for additional paths (env/venv/generated), or set exclude to the default pattern plus your additions, with proper regex grouping/anchoring.
| exclude = "src/metadata/generated|env|venv" | |
| extend-exclude = "src/metadata/generated|env|venv" |
| [tool.pycln] | ||
| all = true | ||
| extend-exclude = "src/metadata/generated" | ||
| exclude = "src/metadata/generated|env|venv" |
There was a problem hiding this comment.
Same as Black: pycln is run against . in nox, and setting exclude here overrides tool defaults. This can cause pycln to scan .nox/virtualenv/build outputs and produce noisy diffs or slow CI. Prefer extend-exclude (or an exclude regex that preserves default exclusions) and consider anchoring/grouping the regex to avoid accidental matches.
| exclude = "src/metadata/generated|env|venv" | |
| extend-exclude = "(^|/)(src/metadata/generated|env|venv)(/|$)" |
|
The Python checkstyle failed. Please run You can install the pre-commit hooks with |
- Updated pyproject.toml to exclude virtual environments from black and pycln. - Standardized import sorting in test_opensearch.py to satisfy checkstyle.
Megh-Shah-08
force-pushed
the
fix/opensearch-aws-session-token-21941
branch
from
66851a6 to
6428120
Compare
Code Review
|
| Compact |
|
Was this helpful? React with 👍 / 👎 | Gitar
|
Describe your changes:
Fixes #21941
Summary: Ingestion was failing when using AWS temporary credentials (Access Key + Secret + Session Token) because the code was attempting to call .get_secret_value() on the awsSessionToken field.
Root Cause: In the awsCredentials.json schema, awsSessionToken is defined as a plain string (without format: password). This means the generated Pydantic model treats it as a standard Python str, which does not have the .get_secret_value() method. This resulted in an AttributeError whenever a session token was provided.
Changes:
Updated connection.py to use awsSessionToken directly as a plain string.
This change aligns the OpenSearch connector with how other AWS-based connectors (like Athena and the base AWS Client) handle this field.
How I tested:
Applied formatting via black.
Type of change:
Checklist:
Fixes #21941: Use awsSessionToken as plain str in OpenSearch connector