[synthrace] Update entities schema #184019

kpatticha · 2024-05-22T13:43:05Z

Summary

Rename assets to entities
Update synthrace to match [EEM][POC] The POC for creating entity-centric indices using entity definitions #183205
Used aggregation to generate the summary metrics as used in data tranform.Timerange is not supported yet, for the rates is used 5 minutes similar to [EEM][POC] The POC for creating entity-centric indices using entity definitions #183205

How to test

Run node scripts/synthtrace.js traces_logs_assets.ts --logLevel debug --clean

Dev tools

GET .entities-synthrace.summary/_search
{}

Produces

{
  "took": 0,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 5,
      "relation": "eq"
    },
    "max_score": 1,
    "hits": [
      {
        "_index": "entities-synthrace.summary",
        "_id": "GYu_uY8B8M9kC_hchEg9",
        "_score": 1,
        "_source": {
          "entity": {
            "id": "apm-only-0",
            "identity": {
              "service": {
                "environment": "Synthtrace: traces_logs_assets",
                "name": "apm-only-0"
              }
            },
            "latestTimestamp": "2024-05-27T11:10:23.765Z",
            "firstSeen": "2024-05-27T11:10:23.765Z",
            "indexPatterns": [
              "metrics-*"
            ],
            "data_stream": {
              "type": [
                "metrics"
              ]
            },
            "agent": {
              "name": "nodejs"
            },
            "metric": {
              "latency": 345500,
              "failedTransactionRate": 0.5,
              "throughput": 0.4
            }
          }
        }
      },
      {
        "_index": "entities-synthrace.summary",
        "_id": "Gou_uY8B8M9kC_hchEg9",
        "_score": 1,
        "_source": {
          "entity": {
            "id": "apm-only-1",
            "identity": {
              "service": {
                "environment": "Synthtrace: traces_logs_assets",
                "name": "apm-only-1"
              }
            },
            "latestTimestamp": "2024-05-27T11:10:23.765Z",
            "firstSeen": "2024-05-27T11:10:23.765Z",
            "indexPatterns": [
              "metrics-*"
            ],
            "data_stream": {
              "type": [
                "metrics"
              ]
            },
            "agent": {
              "name": "nodejs"
            },
            "metric": {
              "latency": 429500,
              "failedTransactionRate": 0.5,
              "throughput": 0.4
            }
          }
        }
      },
      {
        "_index": "entities-synthrace.summary",
        "_id": "G4u_uY8B8M9kC_hchEg9",
        "_score": 1,
        "_source": {
          "entity": {
            "id": "apm-only-2",
            "identity": {
              "service": {
                "environment": "Synthtrace: traces_logs_assets",
                "name": "apm-only-2"
              }
            },
            "latestTimestamp": "2024-05-27T11:10:23.765Z",
            "firstSeen": "2024-05-27T11:10:23.765Z",
            "indexPatterns": [
              "metrics-*"
            ],
            "data_stream": {
              "type": [
                "metrics"
              ]
            },
            "agent": {
              "name": "nodejs"
            },
            "metric": {
              "latency": 1034000,
              "failedTransactionRate": 0.5,
              "throughput": 0.4
            }
          }
        }
      },
      {
        "_index": "entities-synthrace.summary",
        "_id": "HIu_uY8B8M9kC_hchEg9",
        "_score": 1,
        "_source": {
          "entity": {
            "id": "multi-signal-service",
            "identity": {
              "service": {
                "environment": "Synthtrace: traces_logs_assets",
                "name": "multi-signal-service"
              }
            },
            "latestTimestamp": "2024-05-27T11:10:23.765Z",
            "firstSeen": "2024-05-27T11:10:23.765Z",
            "indexPatterns": [
              "metrics-*"
            ],
            "data_stream": {
              "type": [
                "metrics",
                "logs"
              ]
            },
            "agent": {
              "name": "nodejs"
            },
            "metric": {
              "latency": 387500,
              "failedTransactionRate": 0.5,
              "throughput": 0.4,
              "logRatePerMinute": 0.6,
              "logErrorRate": 1
            }
          }
        }
      },
      {
        "_index": "entities-synthrace.summary",
        "_id": "HYu_uY8B8M9kC_hchEg9",
        "_score": 1,
        "_source": {
          "entity": {
            "id": "logs-only-services",
            "identity": {
              "service": {
                "name": "logs-only-services"
              }
            },
            "latestTimestamp": "2024-05-27T11:10:23.765Z",
            "indexPatterns": [
              "logs-*"
            ],
            "data_stream": {
              "type": [
                "logs"
              ]
            },
            "firstSeen": "2024-05-27T11:10:23.765Z",
            "metric": {
              "logRatePerMinute": 0.6,
              "logErrorRate": 0.6666666666666666
            }
          }
        }
      }
    ]
  }
}

apmmachine · 2024-05-22T13:43:19Z

🤖 GitHub comments

Expand to view the GitHub comments

Just comment with:

/oblt-deploy : Deploy a Kibana instance using the Observability test environments.
run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)

elasticmachine · 2024-05-27T11:19:15Z

Pinging @elastic/obs-ux-infra_services-team (Team:obs-ux-infra_services)

…-fix'

kibana-ci · 2024-05-27T13:08:56Z

💔 Build Failed

Buildkite Build
Commit: d50980a
Interpreting CI Failures
Kibana Serverless Image: docker.elastic.co/kibana-ci/kibana-serverless:pr-184019-d50980a01580

Failed CI Steps

Test Failures

[job] [logs] FTR Configs #59 / InfraOps App Metrics UI Hosts View #Single Host Flyout Tabs Overview Tab should show 3 services each with an icon, service name, and url
[job] [logs] FTR Configs #59 / InfraOps App Metrics UI Hosts View #Single Host Flyout Tabs Overview Tab should show 3 services each with an icon, service name, and url

Metrics [docs]

Canvas Sharable Runtime

The Canvas "shareable runtime" is an bundle produced to enable running Canvas workpads outside of Kibana. This bundle is included in third-party webpages that embed canvas and therefor should be as slim as possible.

id	before	after	diff
`module count`	-	5412	+5412
`total size`	-	8.8MB	+8.8MB

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

achyutjhunjhunwala · 2024-05-28T09:13:05Z

packages/kbn-apm-synthtrace/src/lib/entities/entities_synthtrace_es_client.ts

@@ -75,19 +76,27 @@ function getAssetsFilterTransform() {
  });
 }

-function getMergeAssetsTransform() {
-  const mergedDocuments: Record<string, AssetDocument> = {};
+function getMergeEntitesTransform() {


nit: typo in spelling

achyutjhunjhunwala · 2024-05-28T09:30:49Z

packages/kbn-apm-synthtrace/src/lib/utils/create_pivot_transform.ts

+
+  avg({ key }: { key: string }) {
+    const averageMap = new Map();
+    this.trackedValues.forEach((values, key) => {


nit: same name variable collision for key. Also in forEach in the rate function

achyutjhunjhunwala · 2024-05-28T09:43:46Z

packages/kbn-apm-synthtrace/src/lib/utils/create_pivot_transform.ts

+ * Side Public License, v 1.
+ */
+
+class PivotTransform {


Given we are working with Streams here, the usage of name Transform seem misguiding. wdyt ? its actually a factory no ?

achyutjhunjhunwala · 2024-05-28T09:45:13Z

...kbn-apm-synthtrace/src/lib/entities/aggregators/create_traces_service_entities_aggregator.ts

+          'entity.metric': {
+            latency: createPivotTransform(),
+            failedTransactionRate: createPivotTransform(),
+            throughput: createPivotTransform(),


i don't see throughput using any of the record, avg or rate functions. Do you really want it to be a createPivotTransform

achyutjhunjhunwala · 2024-05-28T09:46:46Z

...s/kbn-apm-synthtrace/src/lib/entities/aggregators/create_logs_service_entities_aggregator.ts

+          'entity.data_stream.type': ['logs'],
+          'entity.firstSeen': firstSeen,
+          'entity.metric': {
+            logRatePerMinute: createPivotTransform(),


same as throughput from the other aggregator, logRatePerMinute is not using any of the functions from this function, does it needs to be set here like this ?

achyutjhunjhunwala · 2024-05-28T09:53:06Z

packages/kbn-apm-synthtrace/src/lib/apm/aggregators/create_service_metrics_aggregator.ts

@@ -45,6 +45,9 @@ export function createServiceMetricsAggregator(flushInterval: string) {
            sum: 0,
            value_count: 0,
          },
+          'data_stream.type': 'metrics',


Any specific reason to add these 3 values to all the APM aggregators ?

Concern i see is these 3 values together represent the actual DataStream Name where documents get indexed. And we have the data stream name logic defined in the routing transform here which is different.
packages/kbn-apm-synthtrace/src/lib/apm/client/apm_synthtrace_es_client/get_routing_transform.ts

crespocarlos · 2024-06-19T07:37:38Z

packages/kbn-apm-synthtrace-client/index.ts

@@ -33,4 +33,4 @@ export { generateLongId, generateShortId } from './src/lib/utils/generate_id';
 export { appendHash, hashKeysOf } from './src/lib/utils/hash';
 export type { ESDocumentWithOperation, SynthtraceESAction, SynthtraceGenerator } from './src/types';
 export { log, type LogDocument } from './src/lib/logs';
-export { type AssetDocument } from './src/lib/assets';
+export { type EntityDocument } from './src/lib/assets';


nit

Suggested change

export { type EntityDocument } from './src/lib/assets';

export type { EntityDocument } from './src/lib/assets';

crespocarlos · 2024-06-19T07:42:28Z

packages/kbn-apm-synthtrace/src/lib/entities/aggregators/create_logs_assets_aggregator.ts

@@ -7,6 +7,6 @@
 */


Should the file also be renamed?

crespocarlos · 2024-06-19T07:42:57Z

packages/kbn-apm-synthtrace/src/lib/entities/aggregators/create_logs_assets_aggregator.ts


-export const createLogsAssetsAggregator = createAssetsAggregatorFactory<LogDocument>();
+export const createLogsAssetsAggregator = createEntitiesAggregatorFactory<LogDocument>();


Do we need to rename this const too?

Suggested change

export const createLogsAssetsAggregator = createEntitiesAggregatorFactory<LogDocument>();

export const createLogsEntityAggregator = createEntitiesAggregatorFactory<LogDocument>();

crespocarlos · 2024-06-19T07:43:24Z

packages/kbn-apm-synthtrace/src/lib/entities/aggregators/create_traces_assets_aggregator.ts

@@ -7,6 +7,6 @@
 */


Should the file also be renamed?

crespocarlos · 2024-06-19T07:53:14Z

...s/kbn-apm-synthtrace/src/lib/entities/aggregators/create_logs_service_entities_aggregator.ts

+      const logLevel = event['log.level']!;
+
+      // @ts-expect-error
+      entity['entity.metric'].logErrorRate.record({


Suggestion to remove these ts-expect-error

Suggested change

entity['entity.metric'].logErrorRate.record({

(entity['entity.metric.logErrorRate'] ?? {}) as PivotTransform).logErrorRate.record({

crespocarlos · 2024-06-19T08:30:10Z

packages/kbn-apm-synthtrace/src/scenarios/traces_logs_assets.ts

      const tracesGen = createGeneratorFromArray(traces);
      const tracesGenAssets = createGeneratorFromArray(traces);

      return [
        withClient(
-          assetsEsClient,
+          entitiesEsClient,
          logger.perf('generating_assets_events', () =>


should we change this one too?

Suggested change

logger.perf('generating_assets_events', () =>

logger.perf('generating_entities_events', () =>

crespocarlos · 2024-06-19T08:30:33Z

packages/kbn-apm-synthtrace/src/scenarios/traces_logs_assets.ts

@@ -15,6 +15,7 @@ import {
  log,


Should we rename the scenario too?

crespocarlos · 2024-06-19T08:30:50Z

packages/kbn-apm-synthtrace/src/scenarios/traces_logs_assets.ts

@@ -211,13 +214,13 @@ const scenario: Scenario<ApmFields> = async (runOptions) => {
      const logsGen = createGeneratorFromArray(logsValuesArray);
      const logsGenAssets = createGeneratorFromArray(logsValuesArray);


Suggested change

const logsGenAssets = createGeneratorFromArray(logsValuesArray);

const logsGenEntities = createGeneratorFromArray(logsValuesArray);

crespocarlos · 2024-06-19T08:31:00Z

packages/kbn-apm-synthtrace/src/scenarios/traces_logs_assets.ts

@@ -211,13 +214,13 @@ const scenario: Scenario<ApmFields> = async (runOptions) => {
      const logsGen = createGeneratorFromArray(logsValuesArray);
      const logsGenAssets = createGeneratorFromArray(logsValuesArray);

-      const traces = instances.flatMap((instance) => instanceSpans(instance));
+      const traces = instances.flatMap((instance, index) => instanceSpans(instance, index));
      const tracesGen = createGeneratorFromArray(traces);
      const tracesGenAssets = createGeneratorFromArray(traces);


Suggested change

const tracesGenAssets = createGeneratorFromArray(traces);

const tracesGenEntities = createGeneratorFromArray(traces);

crespocarlos · 2024-06-19T08:31:52Z

x-pack/test/apm_api_integration/tests/asset_services/asset_services.spec.ts

+        entitiesSynthtraceClient.index(
          Readable.from(Array.from(logsGenAssets).concat(Array.from(tracesGenAssets)))
        ),
        logSynthtraceClient.index(logsGen),


Should we rename these too?

Suggested change

entitiesSynthtraceClient.index(

Readable.from(Array.from(logsGenAssets).concat(Array.from(tracesGenAssets)))

),

logSynthtraceClient.index(logsGen),

entitiesSynthtraceClient.index(

Readable.from(Array.from(logsGenEntities).concat(Array.from(tracesGenEntites)))

),

logSynthtraceClient.index(logsGen),

kpatticha added 7 commits May 21, 2024 14:21

Raname assets to entities

de2178b

Rename

1b4cee2

Merge branch 'main' of github.com:elastic/kibana into update-schema-eem

4541fc9

Add data_stream field

51cdd18

Merge entities

c030728

Return agent.name

db4ff05

Fix types

631b5f1

kpatticha added 5 commits May 22, 2024 16:51

Rename

9d0805d

wip

8871800

Merge metrics for the same entities

8d2feb8

Clean up

8dc066e

Update index name

1c616b1

kpatticha marked this pull request as ready for review May 27, 2024 11:19

kpatticha requested review from a team as code owners May 27, 2024 11:19

Merge branch 'main' into update-schema-eem

f90eae7

botelastic bot added ci:project-deploy-observability Create an Observability project Team:obs-ux-infra_services Observability Infrastructure & Services User Experience Team labels May 27, 2024

kpatticha added release_note:skip Skip the PR/issue when compiling release notes v8.15.0 labels May 27, 2024

[CI] Auto-commit changed files from 'node scripts/eslint --no-cache -…

d50980a

…-fix'

achyutjhunjhunwala reviewed May 28, 2024

View reviewed changes

crespocarlos reviewed Jun 19, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[synthrace] Update entities schema #184019

[synthrace] Update entities schema #184019

kpatticha commented May 22, 2024 •

edited

Loading

apmmachine commented May 22, 2024

elasticmachine commented May 27, 2024

kibana-ci commented May 27, 2024 •

edited

Loading

achyutjhunjhunwala May 28, 2024

achyutjhunjhunwala May 28, 2024

achyutjhunjhunwala May 28, 2024

achyutjhunjhunwala May 28, 2024

achyutjhunjhunwala May 28, 2024

achyutjhunjhunwala May 28, 2024

crespocarlos Jun 19, 2024

crespocarlos Jun 19, 2024

crespocarlos Jun 19, 2024

crespocarlos Jun 19, 2024

crespocarlos Jun 19, 2024

crespocarlos Jun 19, 2024

crespocarlos Jun 19, 2024

crespocarlos Jun 19, 2024

crespocarlos Jun 19, 2024

crespocarlos Jun 19, 2024

	export { type EntityDocument } from './src/lib/assets';
	export type { EntityDocument } from './src/lib/assets';


		export const createLogsAssetsAggregator = createAssetsAggregatorFactory<LogDocument>();
		export const createLogsAssetsAggregator = createEntitiesAggregatorFactory<LogDocument>();

	export const createLogsAssetsAggregator = createEntitiesAggregatorFactory<LogDocument>();
	export const createLogsEntityAggregator = createEntitiesAggregatorFactory<LogDocument>();

	entity['entity.metric'].logErrorRate.record({
	(entity['entity.metric.logErrorRate'] ?? {}) as PivotTransform).logErrorRate.record({

	logger.perf('generating_assets_events', () =>
	logger.perf('generating_entities_events', () =>

		@@ -211,13 +214,13 @@ const scenario: Scenario<ApmFields> = async (runOptions) => {
		const logsGen = createGeneratorFromArray(logsValuesArray);
		const logsGenAssets = createGeneratorFromArray(logsValuesArray);

	const logsGenAssets = createGeneratorFromArray(logsValuesArray);
	const logsGenEntities = createGeneratorFromArray(logsValuesArray);

	const tracesGenAssets = createGeneratorFromArray(traces);
	const tracesGenEntities = createGeneratorFromArray(traces);

[synthrace] Update entities schema #184019

Are you sure you want to change the base?

[synthrace] Update entities schema #184019

Conversation

kpatticha commented May 22, 2024 • edited Loading

Summary

How to test

apmmachine commented May 22, 2024

🤖 GitHub comments

elasticmachine commented May 27, 2024

kibana-ci commented May 27, 2024 • edited Loading

💔 Build Failed

Failed CI Steps

Test Failures

Metrics [docs]

Canvas Sharable Runtime

History

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kpatticha commented May 22, 2024 •

edited

Loading

kibana-ci commented May 27, 2024 •

edited

Loading