Fixes 30008: Improve server startup and write-path latency - #30009
Fixes 30008: Improve server startup and write-path latency#30009harshach wants to merge 18 commits into
Conversation
github-actions
Bot
added
backend
safe to test
![gitar-bot[bot]](https://avatars.githubusercontent.com/in/827041?s=60&v=4){kind=small}
✅ PR checks passedThe linked issue has a description and all required Shipping project fields set. Thanks! |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
✅ TypeScript Types Auto-UpdatedThe generated TypeScript types have been automatically updated based on JSON schema changes in this PR. |
Jest test CoverageUI tests summary
|
Code Review ✅ Approved 7 resolved / 7 findingsImproves server startup and write-path performance through fingerprint-gated seeding, parallelized index initialization, and shared classpath scanning. Resolved all identified seed drift and framework qualification issues during the review session. ✅ 7 resolved✅ Bug: CommonUtil.getResources now returns an immutable list
✅ Performance: Search-template gate never stamps fingerprint if any mapping is missing
✅ Quality: ClasspathScanIndex keeps ScanResult open for JVM lifetime
✅ Edge Case: Seed-data gate skips reseeding on DB drift when files are unchanged
✅ Edge Case: Warm-start seed gate reseeds every boot when email config is unset
...and 2 more resolved from earlier reviews OptionsDisplay: compact → Showing less information. Comment with these commands to change the behavior for this request:
Was this helpful? React with 👍 / 👎 | Gitar |
…s-startup-latency # Conflicts: # bootstrap/sql/migrations/native/2.0.0/mysql/schemaChanges.sql # bootstrap/sql/migrations/native/2.0.0/postgres/schemaChanges.sql # openmetadata-service/src/main/java/org/openmetadata/service/events/lifecycle/EntityLifecycleEventDispatcher.java # openmetadata-service/src/main/java/org/openmetadata/service/jdbi3/CollectionDAO.java # openmetadata-service/src/main/java/org/openmetadata/service/search/SearchRepository.java # openmetadata-spec/src/main/resources/json/schema/settings/settings.json # openmetadata-ui/src/main/resources/ui/src/generated/settings/settings.ts
✅ TypeScript Types Auto-UpdatedThe generated TypeScript types have been automatically updated based on JSON schema changes in this PR. |
…s-startup-latency Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
Code Review 👍 Approved with suggestions 10 resolved / 12 findingsImproves server startup performance and write-path latency through fingerprint-gated seed reconciliation, shared classpath scanning, and direct JSON conversion. Consider refining the JSON exception mapping and record-batch timestamp tie handling to prevent potential 412 error misclassifications and duplicate rows. 💡 Quality: JsonException mapped to misleading concurrent-change 412 message📄 openmetadata-service/src/main/java/org/openmetadata/service/jdbi3/SystemRepository.java:427-431 A JsonException from JsonUtils.applyPatch is typically raised for an inapplicable/malformed patch (e.g. a failing test op or a replace on a missing path), not a concurrent modification. Mapping it to PreconditionFailedException with the message "Glossary term relation settings changed while the JSON Patch was being applied" returns HTTP 412 with a cause that misleads clients who actually sent a bad patch. Consider distinguishing patch-application failures (400 Bad Request with a patch-specific message) from the genuine optimistic-concurrency case handled at the updatedRows==0 check. 💡 Edge Case: getLatestRecordBatch can return duplicate rows on timestamp ties📄 openmetadata-service/src/main/java/org/openmetadata/service/jdbi3/CollectionDAO.java:10293-10305 The new ✅ 10 resolved✅ Bug: CommonUtil.getResources now returns an immutable list
✅ Performance: Search-template gate never stamps fingerprint if any mapping is missing
✅ Quality: ClasspathScanIndex keeps ScanResult open for JVM lifetime
✅ Edge Case: Seed-data gate skips reseeding on DB drift when files are unchanged
✅ Edge Case: Warm-start seed gate reseeds every boot when email config is unset
...and 5 more resolved from earlier reviews 🤖 Prompt for agentsOptionsDisplay: compact → Showing less information. Comment with these commands to change the behavior for this request:
Was this helpful? React with 👍 / 👎 | Gitar |
| } catch (Exception exception) { | ||
| failed++; | ||
| LOG.warn("Failed to create index template for {}", entry.getKey(), e); | ||
| LOG.warn( | ||
| "Failed to create index template for {}: {}", entry.getKey(), exception.getMessage()); |
|
Describe your changes:
Fixes #30008
I improved server startup, request/write latency, and lineage query performance because warm boots repeated bundled-data and search reconciliation, writes repeated JSON serialization, and the pipeline-summary service type was interpolated into SQL.
Type of change:
High-level design:
2.0.0-SNAPSHOT, so no production migration is required, but development databases that already seeded flat control FQNs should be resetTests:
Use cases covered
Unit tests
CommonUtilTest,StartupTimerTest,SeedDataGateTest,FrameworkSeedLoaderTest,ClasspathScanIndexTest,CollectionRegistryTest,RBACConditionEvaluatorCacheTest,IndexTemplateManagerTest,EntityRepositoryStorageJsonReuseTest,EntityLifecycleEventDispatcherTest,RequestLatencyContextTest,MicrometerBundleTest,SettingsRowMapperTest, andJsonUtilsTestSeedDataGateTestandFrameworkSeedLoaderTestpassed 9/9 combined;IndexTemplateManagerTestpassed 10/10Backend integration tests
openmetadata-integration-tests/for new/changed backend behavior.PipelineResourceIT.java,LineageResourceIT.java,SeedDataPresenceIT.java,AIFrameworkControlResourceIT.java, andIndexTemplateIT.javaIngestion integration tests
Playwright (UI) tests
Manual testing performed
mvn spotless:apply,mvn spotless:check, schema validation, andgit diff --checkUI screen recording / screenshots:
Not applicable.
Checklist:
Fixes <issue-number>: <short explanation>Fixes #<issue-number>above.Greptile Summary
This PR improves startup gating, search setup, write latency, and lineage query performance. The main changes are:
Confidence Score: 5/5
This looks safe to merge.
Important Files Changed
Reviews (17): Last reviewed commit: "Merge branch 'main' into harshach/perf-a..." | Re-trigger Greptile
Context used (3)