feat(metric): add CSV import/export and native DAX expression language (#27474)#27547
feat(metric): add CSV import/export and native DAX expression language (#27474)#27547RajdeepKushwaha5 wants to merge 3 commits intoopen-metadata:mainfrom
Conversation
open-metadata#27474) Adds CSV import/export endpoints for the Metric entity (mirroring the existing pattern used by Database/Worksheet/TestCase) and promotes DAX from a Custom workaround to a first-class MetricExpressionLanguage enum value. Endpoints: GET /v1/metrics/name/{name}/export, GET /v1/metrics/name/{name}/exportAsync, PUT /v1/metrics/name/{name}/import, PUT /v1/metrics/name/{name}/importAsync. Use * for platform-wide or a Domain FQN to scope. Auto-sync of DAX measures from upstream PowerBI ingestion remains as follow-up work.
|
Hi there 👋 Thanks for your contribution! The OpenMetadata team will review the PR shortly! Once it has been labeled as Let us know if you need any help! |
![gitar-bot[bot]](https://avatars.githubusercontent.com/in/827041?s=60&v=4){kind=small}
There was a problem hiding this comment.
Pull request overview
This PR adds bulk CSV import/export support for Metrics in the OpenMetadata service and promotes DAX to a first-class MetricExpressionLanguage in the Metric schema, enabling round-tripping of DAX expressions through schema/JSON/CSV.
Changes:
- Added
DAXto Metric expression language enum in the spec schema. - Introduced Metric CSV documentation (15-column header) under service JSON data resources.
- Implemented Metric CSV import/export in
MetricRepositoryand exposed new REST endpoints inMetricResource(sync + async).
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 5 comments.
| File | Description |
|---|---|
| openmetadata-spec/src/main/resources/json/schema/entity/data/metric.json | Adds DAX enum value for metricExpression.language (schema + javaEnums). |
| openmetadata-service/src/main/resources/json/data/metric/metricCsvDocumentation.json | New CSV documentation describing Metric CSV format and columns. |
| openmetadata-service/src/main/java/org/openmetadata/service/jdbi3/MetricRepository.java | Adds CSV export/import implementation via EntityCsv pattern for metrics. |
| openmetadata-service/src/main/java/org/openmetadata/service/resources/metrics/MetricResource.java | Adds REST endpoints for CSV export/import (sync + async). |
| private List<Metric> getMetricsForExport(String name) { | ||
| Fields fields = new Fields(allowedFields, "owners,reviewers,relatedMetrics,tags,domains"); | ||
| if (name == null || name.isBlank() || "*".equals(name)) { | ||
| return listAll(fields, new ListFilter(NON_DELETED)); | ||
| } | ||
| // Otherwise, treat name as a domain FQN and filter to metrics owned by that domain | ||
| ListFilter filter = new ListFilter(NON_DELETED).addQueryParam("domain", name); | ||
| return listAll(fields, filter); | ||
| } |
There was a problem hiding this comment.
getMetricsForExport adds query param domain, but ListFilter's domain filter is driven by domainId (see ListFilter#getDomainCondition). As written, passing a Domain FQN to /name/{name}/export won't actually scope the export. Resolve the Domain (by FQN) to its UUID and set domainId on the filter (or use the same filtering mechanism as other domain-scoped queries).
| @Override | ||
| public CsvImportResult importFromCsv( | ||
| String name, String csv, boolean dryRun, String user, boolean recursive) throws IOException { | ||
| return new MetricCsv(user).importCsv(csv, dryRun); | ||
| } | ||
|
|
||
| @Override | ||
| public CsvImportResult importFromCsv( | ||
| String name, | ||
| String csv, | ||
| boolean dryRun, | ||
| String user, | ||
| boolean recursive, | ||
| CsvImportProgressCallback callback) | ||
| throws IOException { | ||
| return new MetricCsv(user).importCsv(csv, dryRun); | ||
| } |
There was a problem hiding this comment.
Both importFromCsv overloads ignore the name parameter entirely. This means /name/{name}/import cannot implement the documented semantics (platform-wide * vs Domain-scoped import), and imported metrics won’t be automatically associated with the domain specified in the path. Use name to resolve a target Domain (when not *) and apply it consistently during import (e.g., default/override domains when the CSV column is empty).
| @Operation( | ||
| operationId = "importMetricsAsync", | ||
| summary = "Import metrics from CSV asynchronously", | ||
| description = | ||
| "Import metrics from CSV asynchronously. Returns a job id that can be polled for completion.", | ||
| responses = { | ||
| @ApiResponse( | ||
| responseCode = "200", | ||
| description = "Import initiated successfully", | ||
| content = | ||
| @Content( | ||
| mediaType = "application/json", | ||
| schema = @Schema(implementation = CsvImportResult.class))) | ||
| }) |
There was a problem hiding this comment.
importCsvAsync is documented as returning a CsvImportResult, but importCsvInternalAsync actually returns a job response (CSVImportResponse with jobId/message). Update the OpenAPI response schema to match the real response type so clients don’t deserialize the async response incorrectly.
| String metricName = csvRecord.get(0); | ||
| // Metric FQN is just the name (no parent in current schema) | ||
| Metric metric; | ||
| try { | ||
| metric = Entity.getEntityByName(METRIC, metricName, "*", Include.NON_DELETED); | ||
| } catch (EntityNotFoundException ex) { | ||
| LOG.debug("Metric not found: {}, it will be created during import.", metricName); | ||
| metric = new Metric().withName(metricName).withFullyQualifiedName(metricName); | ||
| } |
There was a problem hiding this comment.
During CSV import, each row calls Entity.getEntityByName(METRIC, metricName, "*", ...), which fetches the full Metric payload even though the import path only needs an existence check / id for update detection. For large imports this adds unnecessary per-row overhead; consider using a lightweight lookup (e.g., repository findByNameOrNull / findMatchForImport) and building the Metric from CSV fields, letting the base EntityCsv#createEntity handle create-vs-update.
…y FQN, scope import
- importFromCsv(callback) now passes the CsvImportProgressCallback through to MetricCsv.importCsv so async imports report progress (was: callback dropped).
- getMetricsForExport now resolves the {name} path parameter as a Domain FQN to its UUID and filters via domainId, matching ListFilter#getDomainCondition. Previously the unrecognized 'domain' query param was silently ignored and exports returned every metric.
- importFromCsv now also resolves {name} to a Domain reference and applies it as the default domain on imported rows that leave the domains column blank, honoring the documented endpoint semantics.
|
Hi there 👋 Thanks for your contribution! The OpenMetadata team will review the PR shortly! Once it has been labeled as Let us know if you need any help! |
- MetricCsv.createEntity now uses findByNameOrNull instead of Entity.getEntityByName with all fields, avoiding a per-row fetch of every related field. The CSV row is the source of truth for all writable fields anyway. - importMetricsAsync OpenAPI annotation now declares CSVImportResponse (jobId/message) instead of CsvImportResult, matching what importCsvInternalAsync actually returns.
|
Hi there 👋 Thanks for your contribution! The OpenMetadata team will review the PR shortly! Once it has been labeled as Let us know if you need any help! |
Code Review ✅ Approved 1 resolved / 1 findingsImplements CSV import/export and native DAX expression language functionality. The CsvImportProgressCallback parameter ignore issue has been resolved. ✅ 1 resolved✅ Bug: importFromCsv ignores CsvImportProgressCallback parameter
OptionsDisplay: compact → Showing less information. Comment with these commands to change:
Was this helpful? React with 👍 / 👎 | Gitar |
| @GET | ||
| @Path("/name/{name}/export") | ||
| @Produces({MediaType.TEXT_PLAIN + "; charset=UTF-8"}) | ||
| @Valid | ||
| @Operation( |
There was a problem hiding this comment.
The new Metrics CSV import/export endpoints should be covered by integration tests. There is already a MetricResourceIT (extends BaseEntityIT) with a reusable CSV import/export test suite gated by supportsImportExport; please enable that for Metrics and implement the required overrides/CSV generators so export, dry-run import, and round-trip import are validated (including * vs Domain-FQN scoping).
Summary
Resolves the first two asks of #27474 — Metrics CSV Import/Export & DAX Measure Linking:
MetricExpressionLanguage— DAX measures no longer need to be carried as a "Custom" workaround; they round-trip through the schema, JSON, and CSV.The third ask — auto-syncing OpenMetadata Metrics with upstream PowerBI DAX measures — is intentionally out of scope for this PR (see Follow-up Work below).
What changed
openmetadata-spec/.../entity/data/metric.json— added"DAX"to both theenumarray and thejavaEnumsofmetricExpression.language(betweenPythonandExternal).openmetadata-service/.../json/data/metric/metricCsvDocumentation.json— new file describing the 15-column CSV header.MetricRepository.java— implements the standardEntityCsvpattern (getMetricsForExport,getMetricsCsv, plus inner classMetricCsv extends EntityCsv<Metric>) used elsewhere by Worksheet / TestCase / Database.MetricResource.java— addsGET /name/{name}/export,GET /name/{name}/exportAsync,PUT /name/{name}/import,PUT /name/{name}/importAsync.CSV columns (in order)
name(required) ·displayName·description·metricType·unitOfMeasurement·customUnitOfMeasurement·granularity·expressionLanguage·expressionCode·relatedMetrics·owners·reviewers·tags·glossaryTerms·domainsexpressionLanguageaccepts:SQL,Java,JavaScript,Python,DAX,External.Endpoints
{name}accepts:*— platform-wide (all metrics, no domain filter on export; new metrics created without a domain on import).The semantics mirror the existing CSV endpoints on
Database/Glossary/TestCase/Worksheet.Why
The issue (#27474) calls out two friction points that this PR removes:
Follow-up work (not in this PR)
Auto-syncing a Metric definition when its upstream PowerBI DAX measure changes is a multi-week effort that touches:
model.bim/dataset.jsonto surface DAX measures.dashboardDataModel→metric.This will be tracked as a separate issue and PR so the CSV/DAX changes can land independently.
Testing
mvn -pl openmetadata-spec install -DskipTests—MetricExpressionLanguage.DAXis generated.mvn -pl openmetadata-service -am compile -DskipTests— zero errors inMetricRepository.javaandMetricResource.java.mvn spotless:apply— formatting clean.Files changed
Issue link
Closes part of #27474 (CSV import/export + native DAX language). The DAX-measure auto-sync portion will be tracked separately.