Conversation
github-actions
bot
added
Ingestion
safe to test
![gitar-bot[bot]](https://avatars.githubusercontent.com/in/827041?s=60&v=4){kind=small}
There was a problem hiding this comment.
Pull request overview
Enhances dbt HTTP artifact fetching in the ingestion framework by allowing callers to configure request headers and SSL verification, improving support for private artifact endpoints (e.g., GitLab/GitHub) and custom CA validation.
Changes:
- Extends
DbtHttpConfigJSON schema withdbtHttpHeaders,dbtVerifySSL, anddbtSSLConfig. - Updates dbt HTTP fetching to forward headers and SSL verification options to all
requests.get()calls, with improved manifest-fetch error messaging for SSL/auth failures. - Adds unit tests validating header/SSL forwarding and key error paths.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| openmetadata-spec/src/main/resources/json/schema/metadataIngestion/dbtconfig/dbtHttpConfig.json | Adds schema fields to configure HTTP headers and SSL verification for dbt HTTP artifact downloads. |
| ingestion/src/metadata/ingestion/source/database/dbt/dbt_config.py | Applies configured headers/SSL settings to dbt artifact HTTP requests and improves certain error messages. |
| ingestion/tests/unit/test_dbt_http_config.py | Introduces unit tests for new dbt HTTP headers/SSL behavior and error handling. |
✅ TypeScript Types Auto-UpdatedThe generated TypeScript types have been automatically updated based on JSON schema changes in this PR. |
🟡 Playwright Results — all passed (25 flaky)✅ 3593 passed · ❌ 0 failed · 🟡 25 flaky · ⏭️ 207 skipped
🟡 25 flaky test(s) (passed on retry)
How to debug locally# Download playwright-test-results-<shard> artifact and unzip
npx playwright show-trace path/to/trace.zip # view trace |
| "additionalProperties": { | ||
| "type": "string" |
There was a problem hiding this comment.
dbtHttpHeaders will commonly include secrets (e.g., Authorization, PRIVATE-TOKEN). The schema currently defines it as a plain object of strings without mask: true/secret formatting, which can cause credentials to be exposed via persisted pipeline configs, API responses, or UI rendering. Consider marking this property as masked (or modelling header values as secret strings) so tokens are not displayed or logged inadvertently.
| "additionalProperties": { | |
| "type": "string" | |
| "mask": true, | |
| "additionalProperties": { | |
| "type": "string", | |
| "format": "password", | |
| "mask": true |
Jest test CoverageUI tests summary
|
Code Review ✅ Approved 1 resolved / 1 findingsFixes HTTP config parameters in dbt by patching requests.get globally instead of at import location, improving test reliability. No issues found. ✅ 1 resolved✅ Quality: Tests patch requests.get globally instead of at import location
OptionsDisplay: compact → Showing less information. Comment with these commands to change:
Was this helpful? React with 👍 / 👎 | Gitar |
|
|
Describe your changes:
Fixes #22942
This pull request introduces enhanced support for custom HTTP headers and SSL verification when fetching dbt artifacts via HTTP, improves error handling, and adds comprehensive unit tests for these features. These changes make it easier and more secure to connect to dbt artifact endpoints that require authentication (such as private GitLab or GitHub repositories) or custom SSL settings.
dbt HTTP configuration and ingestion improvements:
dbtHttpHeaders) and SSL verification options (dbtVerifySSL,dbtSSLConfig) to theDbtHttpConfigschema and ingestion logic, allowing users to specify authentication headers and SSL/TLS settings for all HTTP requests when fetching dbt artifacts. [1] [2] [3] [4] [5] [6]Testing improvements:
test_dbt_http_config.py) to verify that custom headers and SSL settings are correctly applied to all HTTP requests and that errors are handled as expected.These changes significantly improve the flexibility, security, and reliability of dbt ingestion.
Type of change:
Checklist:
Fixes <issue-number>: <short explanation>