Fix #27079: [BUG] S3 Ingestion Failure: Pydantic ValidationError 'ext… by harshach · Pull Request #27180 · open-metadata/OpenMetadata

harshach · 2026-04-08T19:36:23Z

…ra_forbidden' for BucketArn in S3BucketResponse

Describe your changes:

Fixes

I worked on ... because ...

Type of change:

Checklist:

I have read the CONTRIBUTING document.
My PR title is Fixes <issue-number>: <short explanation>
I have commented on my code, particularly in hard-to-understand areas.
For JSON Schema changes: I updated the migration scripts or explained why it is not needed.

Summary by Gitar

Fixed validation error:
- Changed Pydantic extra config from forbid to ignore in S3BucketResponse, GCSBucketResponse, and SageMakerModel to handle extra fields like BucketArn and BucketRegion
Updated test data:
- Added BucketArn and BucketRegion fields to S3 bucket mock responses in test cases

_{This will update automatically on new commits.}

…ra_forbidden' for BucketArn in S3BucketResponse

Copilot

Pull request overview

This PR addresses ingestion-time Pydantic ValidationError failures caused by cloud SDK responses including additional (previously forbidden) fields, primarily for S3 bucket listing responses.

Changes:

Relaxed Pydantic models to ignore unexpected fields in S3 bucket responses (and similarly for GCS bucket + SageMaker model response models).
Updated S3 unit test fixtures to include additional bucket fields (e.g., BucketArn, BucketRegion) to reproduce/guard the regression.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

File	Description
ingestion/tests/unit/topology/storage/test_s3_storage.py	Extends mocked S3 `list_buckets` response with extra fields to validate tolerant parsing.
ingestion/src/metadata/ingestion/source/storage/s3/models.py	Changes `S3BucketResponse` to ignore extra keys to prevent `extra_forbidden` failures.
ingestion/src/metadata/ingestion/source/storage/gcs/models.py	Migrates `GCSBucketResponse` config to Pydantic v2 `ConfigDict` and ignores extra keys.
ingestion/src/metadata/ingestion/source/mlmodel/sagemaker/metadata.py	Updates `SageMakerModel` to ignore extra keys.

Copilot · 2026-04-08T19:39:52Z

ingestion/src/metadata/ingestion/source/storage/gcs/models.py

 from typing import List, Optional

-from pydantic import BaseModel, Extra, Field
+from pydantic import BaseModel, ConfigDict, Field


Extra was removed from the pydantic imports, but this module still references Extra in GCSContainerDetails.Config later in the file. This will raise a NameError at import time and break GCS ingestion/tests. Re-add the Extra import or (preferred, Pydantic v2 style) migrate GCSContainerDetails to model_config = ConfigDict(extra="forbid") and drop the inner Config class.

cc @harshach

TeddyCr · 2026-04-08T19:44:36Z

ingestion/src/metadata/ingestion/source/mlmodel/sagemaker/metadata.py

 class SageMakerModel(BaseModel):
    model_config = ConfigDict(
-        extra="forbid",
+        extra="ignore",


ignore is the default so we can just delete that completely

gitar-bot · 2026-04-08T19:49:41Z

Code Review ✅ Approved

Fixes S3 ingestion failure caused by Pydantic ValidationError. No issues found.

Options

Display: compact → Showing less information.

Comment with these commands to change:

`Compact`
`gitar display:verbose`

_{Was this helpful? React with 👍 / 👎 | Gitar}

github-actions · 2026-04-08T21:42:07Z

🟡 Playwright Results — all passed (25 flaky)

✅ 3593 passed · ❌ 0 failed · 🟡 25 flaky · ⏭️ 207 skipped

Shard	Passed	Flaky	Skipped
🟡 Shard 1	453	4	2
🟡 Shard 2	639	3	32
🟡 Shard 3	647	4	26
🟡 Shard 4	618	4	47
🟡 Shard 5	606	1	67
🟡 Shard 6	630	9	33

🟡 25 flaky test(s) (passed on retry)

Features/DataAssetRulesDisabled.spec.ts › Verify the Database Service entity item action after rules disabled (shard 1, 1 retry)
Features/CustomizeDetailPage.spec.ts › Stored Procedure - customization should work (shard 1, 1 retry)
Features/CustomizeDetailPage.spec.ts › Glossary - customization should work (shard 1, 1 retry)
Pages/UserCreationWithPersona.spec.ts › Create user with persona and verify on profile (shard 1, 1 retry)
Features/BulkEditEntity.spec.ts › Glossary (shard 2, 1 retry)
Features/BulkImport.spec.ts › Keyboard Delete selection (shard 2, 1 retry)
Features/DomainTierCertificationVoting.spec.ts › DataProduct - Certification assign, update, and remove (shard 2, 1 retry)
Features/Permissions/GlossaryPermissions.spec.ts › Team-based permissions work correctly (shard 3, 1 retry)
Flow/ExploreDiscovery.spec.ts › Should display deleted assets when showDeleted is checked and deleted is not present in queryFilter (shard 3, 1 retry)
Flow/PersonaDeletionUserProfile.spec.ts › User profile loads correctly before and after persona deletion (shard 3, 1 retry)
Flow/PersonaFlow.spec.ts › Set default persona for team should work properly (shard 3, 1 retry)
Pages/Customproperties-part2.spec.ts › entityReferenceList shows item count, scrollable list, no expand toggle (shard 4, 1 retry)
Pages/DescriptionVisibility.spec.ts › Customized Table detail page Description widget shows long description (shard 4, 1 retry)
Pages/Entity.spec.ts › Glossary Term Add, Update and Remove (shard 4, 1 retry)
Pages/Entity.spec.ts › User should be denied access to edit description when deny policy rule is applied on an entity (shard 4, 1 retry)
Pages/ExploreTree.spec.ts › Verify Database and Database Schema available in explore tree (shard 5, 1 retry)
Pages/Lineage/LineageFilters.spec.ts › Verify lineage schema filter selection (shard 6, 1 retry)
Pages/Login.spec.ts › Refresh should work (shard 6, 2 retries)
Pages/ODCSImportExport.spec.ts › Multi-object ODCS contract - object selector shows all schema objects (shard 6, 1 retry)
Pages/ProfilerConfigurationPage.spec.ts › Non admin user (shard 6, 1 retry)
Pages/ServiceListing.spec.ts › should render the service listing page (shard 6, 1 retry)
Pages/Tag.spec.ts › Verify Owner Add Delete (shard 6, 1 retry)
Pages/Users.spec.ts › Permissions for table details page for Data Consumer (shard 6, 1 retry)
Pages/Users.spec.ts › Should add, remove, and navigate to persona pages for Personas section (shard 6, 1 retry)
VersionPages/EntityVersionPages.spec.ts › Directory (shard 6, 1 retry)

📦 Download artifacts

How to debug locally

# Download playwright-test-results-<shard> artifact and unzip
npx playwright show-trace path/to/trace.zip    # view trace

github-actions · 2026-04-09T00:37:37Z

Failed to cherry-pick changes to the 1.12.5 branch.
Please cherry-pick the changes manually.
You can find more details here.

sonarqubecloud · 2026-04-09T01:39:54Z

Quality Gate passed for 'open-metadata-ingestion'

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
100.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

#27180) * Fix #27079: [BUG] S3 Ingestion Failure: Pydantic ValidationError 'extra_forbidden' for BucketArn in S3BucketResponse * Address comments

…ionError 'ext… (open-metadata#27180) * Fix open-metadata#27079: [BUG] S3 Ingestion Failure: Pydantic ValidationError 'extra_forbidden' for BucketArn in S3BucketResponse * Address comments

Fix #27079: [BUG] S3 Ingestion Failure: Pydantic ValidationError 'ext…

228e779

…ra_forbidden' for BucketArn in S3BucketResponse

harshach requested a review from a team as a code owner April 8, 2026 19:36

Copilot AI review requested due to automatic review settings April 8, 2026 19:36

github-actions bot added backend safe to test Add this label to run secure Github workflows on PRs labels Apr 8, 2026

Copilot started reviewing on behalf of harshach April 8, 2026 19:37 View session

Copilot AI reviewed Apr 8, 2026

View reviewed changes

TeddyCr reviewed Apr 8, 2026

View reviewed changes

harshach had a problem deploying to test April 8, 2026 19:47 — with GitHub Actions Error

Address comments

9a7fa2c

harshach temporarily deployed to test April 8, 2026 19:58 — with GitHub Actions Inactive

TeddyCr approved these changes Apr 8, 2026

View reviewed changes

harshach added the To release Will cherry-pick this PR into the release branch label Apr 9, 2026

harshach merged commit 2089b6a into main Apr 9, 2026
57 checks passed

harshach deleted the S3_list_apis branch April 9, 2026 00:36

harshach temporarily deployed to test April 9, 2026 00:46 — with GitHub Actions Inactive

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix #27079: [BUG] S3 Ingestion Failure: Pydantic ValidationError 'ext…#27180

Fix #27079: [BUG] S3 Ingestion Failure: Pydantic ValidationError 'ext…#27180
harshach merged 2 commits intomainfrom
S3_list_apis

harshach commented Apr 8, 2026 •

edited by gitar-bot bot

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Apr 8, 2026

Uh oh!

TeddyCr Apr 8, 2026

Uh oh!

TeddyCr Apr 8, 2026

Uh oh!

gitar-bot bot commented Apr 8, 2026 •

edited

Loading

Uh oh!

github-actions bot commented Apr 8, 2026 •

edited

Loading

Uh oh!

Uh oh!

github-actions bot commented Apr 9, 2026

Uh oh!

sonarqubecloud bot commented Apr 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

harshach commented Apr 8, 2026 • edited by gitar-bot bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Describe your changes:

Type of change:

Checklist:

Summary by Gitar

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Apr 8, 2026

Choose a reason for hiding this comment

Uh oh!

TeddyCr Apr 8, 2026

Choose a reason for hiding this comment

Uh oh!

TeddyCr Apr 8, 2026

Choose a reason for hiding this comment

Uh oh!

gitar-bot bot commented Apr 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Apr 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🟡 Playwright Results — all passed (25 flaky)

Uh oh!

Uh oh!

github-actions bot commented Apr 9, 2026

Uh oh!

sonarqubecloud bot commented Apr 9, 2026

Quality Gate passed for 'open-metadata-ingestion'

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

harshach commented Apr 8, 2026 •

edited by gitar-bot bot

Loading

gitar-bot bot commented Apr 8, 2026 •

edited

Loading

github-actions bot commented Apr 8, 2026 •

edited

Loading