feat(meta) Add configuration for meta tables in generic metrics

[evanh]

This adds all the YAML files for the storages and entities required to query
the meta tables. Note that this is 4 copies of the same files, just replacing
the metric type in each one (counters/sets/gauges/distributions).

A required column for the meta table is the use case ID, but that is a string
column. This also slightly modifies the validator to allow string columns as
well as integer columns. The intent was to keep the strictness of the original
check (which specifically looks for ints), so this added a new similar check.

For reviewers: This is a big PR, but there are really only 4 new files added.
So just reviewing the files for counters would be sufficient.

[codecov]

Test Failures Detected: Due to failing tests, we cannot provide coverage reports at this time.

❌ Failed Test Results:

Completed 840 tests with 1 failed, 837 passed and 2 skipped.

View the full list of failed tests

Test Description

Failure message

Testsuite: pytest Test name: tests.datasets.test_entity_factory::test_all_names Envs: - default

Traceback (most recent call last): File ".../tests/datasets/test_entity_factory.py", line 67, in test_all_names assert set(get_all_entity_names()) == set(ENTITY_KEYS) AssertionError: assert {EntityKey.GENERIC_METRICS_DISTRIBUTIONS_META_TAG_VALUES,\n EntityKey.GENERIC_METRICS_GAUGES_META_TAG_VALUES,\n EntityKey.GENERIC_METRICS_COUNTERS_META,\n EntityKey.GENERIC_METRICS_GAUGES,\n EntityKey.REPLAYS,\n EntityKey.GENERIC_METRICS_COUNTERS_META_TAG_VALUES,\n EntityKey.GENERIC_METRICS_GAUGES_META,\n EntityKey.SEARCH_ISSUES,\n EntityKey.GENERIC_METRICS_DISTRIBUTIONS_META,\n EntityKey.GENERIC_METRICS_DISTRIBUTIONS,\n EntityKey.GROUP_ATTRIBUTES,\n EntityKey.METRICS_SUMMARIES,\n EntityKey.GENERIC_METRICS_SETS,\n EntityKey.OUTCOMES,\n EntityKey.SPANS,\n EntityKey.GENERIC_METRICS_SETS_META_TAG_VALUES,\n EntityKey.GENERIC_METRICS_COUNTERS,\n EntityKey.PROFILES,\n EntityKey.METRICS_COUNTERS,\n EntityKey.GROUPASSIGNEE,\n EntityKey.__INTERNAL_GENERIC_ORG_METRICS_DISTRIBUTIONS,\n EntityKey.METRICS_SETS,\n EntityKey.GENERIC_METRICS_SETS_META,\n EntityKey.ORG_SESSIONS,\n EntityKey.EVENTS,\n EntityKey.METRICS_DISTRIBUTIONS,\n EntityKey.GENERIC_ORG_METRICS_COUNTERS,\n EntityKey.DISCOVER_EVENTS,\n EntityKey.OUTCOMES_RAW,\n EntityKey.TRANSACTIONS,\n EntityKey.SESSIONS,\n EntityKey.DISCOVER_TRANSACTIONS,\n EntityKey.FUNCTIONS,\n EntityKey.ORG_METRICS_COUNTERS,\n EntityKey.__INTERNAL_GENERIC_ORG_METRICS_SETS,\n EntityKey.DISCOVER,\n EntityKey.__INTERNAL_ORG_METRICS_DISTRIBUTIONS,\n EntityKey.__INTERNAL_ORG_METRICS_SETS,\n EntityKey.GROUPEDMESSAGE} == {EntityKey.EVENTS,\n EntityKey.DISCOVER,\n EntityKey.SPANS,\n EntityKey.METRICS_COUNTERS,\n EntityKey.GROUP_ATTRIBUTES,\n EntityKey.DISCOVER_EVENTS,\n EntityKey.METRICS_DISTRIBUTIONS,\n EntityKey.METRICS_SETS,\n EntityKey.PROFILES,\n EntityKey.GENERIC_ORG_METRICS_COUNTERS,\n EntityKey.FUNCTIONS,\n EntityKey.METRICS_SUMMARIES,\n EntityKey.REPLAYS,\n EntityKey.__INTERNAL_ORG_METRICS_SETS,\n EntityKey.TRANSACTIONS,\n EntityKey.GENERIC_METRICS_SETS,\n EntityKey.DISCOVER_TRANSACTIONS,\n EntityKey.__INTERNAL_GENERIC_ORG_METRICS_DISTRIBUTIONS,\n EntityKey.GENERIC_METRICS_COUNTERS,\n EntityKey.__INTERNAL_ORG_METRICS_DISTRIBUTIONS,\n EntityKey.GROUPASSIGNEE,\n EntityKey.ORG_METRICS_COUNTERS,\n EntityKey.__INTERNAL_GENERIC_ORG_METRICS_SETS,\n EntityKey.ORG_SESSIONS,\n EntityKey.GENERIC_METRICS_DISTRIBUTIONS,\n EntityKey.GENERIC_METRICS_GAUGES,\n EntityKey.OUTCOMES,\n EntityKey.GROUPEDMESSAGE,\n EntityKey.SESSIONS,\n EntityKey.SEARCH_ISSUES,\n EntityKey.OUTCOMES_RAW} Extra items in the left set: EntityKey.GENERIC_METRICS_GAUGES_META_TAG_VALUES EntityKey.GENERIC_METRICS_SETS_META_TAG_VALUES EntityKey.GENERIC_METRICS_COUNTERS_META_TAG_VALUES EntityKey.GENERIC_METRICS_DISTRIBUTIONS_META EntityKey.GENERIC_METRICS_GAUGES_META EntityKey.GENERIC_METRICS_COUNTERS_META EntityKey.GENERIC_METRICS_DISTRIBUTIONS_META_TAG_VALUES EntityKey.GENERIC_METRICS_SETS_META Full diff: { + EntityKey.GENERIC_METRICS_DISTRIBUTIONS_META_TAG_VALUES, + EntityKey.GENERIC_METRICS_GAUGES_META_TAG_VALUES, + EntityKey.GENERIC_METRICS_COUNTERS_META, + EntityKey.GENERIC_METRICS_GAUGES, + EntityKey.REPLAYS, + EntityKey.GENERIC_METRICS_COUNTERS_META_TAG_VALUES, + EntityKey.GENERIC_METRICS_GAUGES_META, + EntityKey.SEARCH_ISSUES, + EntityKey.GENERIC_METRICS_DISTRIBUTIONS_META, + EntityKey.GENERIC_METRICS_DISTRIBUTIONS, + EntityKey.GROUP_ATTRIBUTES, + EntityKey.METRICS_SUMMARIES, + EntityKey.GENERIC_METRICS_SETS, + EntityKey.OUTCOMES, + EntityKey.SPANS, + EntityKey.GENERIC_METRICS_SETS_META_TAG_VALUES, + EntityKey.GENERIC_METRICS_COUNTERS, + EntityKey.PROFILES, + EntityKey.METRICS_COUNTERS, + EntityKey.GROUPASSIGNEE, + EntityKey.__INTERNAL_GENERIC_ORG_METRICS_DISTRIBUTIONS, + EntityKey.METRICS_SETS, + EntityKey.GENERIC_METRICS_SETS_META, + EntityKey.ORG_SESSIONS, EntityKey.EVENTS, + EntityKey.METRICS_DISTRIBUTIONS, - EntityKey.DISCOVER, - EntityKey.SPANS, - EntityKey.METRICS_COUNTERS, + EntityKey.GENERIC_ORG_METRICS_COUNTERS, & ++++++++++++ - EntityKey.GROUP_ATTRIBUTES, EntityKey.DISCOVER_EVENTS, - EntityKey.METRICS_DISTRIBUTIONS, - EntityKey.METRICS_SETS, - EntityKey.PROFILES, - EntityKey.GENERIC_ORG_METRICS_COUNTERS, + EntityKey.OUTCOMES_RAW, + EntityKey.TRANSACTIONS, + EntityKey.SESSIONS, + EntityKey.DISCOVER_TRANSACTIONS, EntityKey.FUNCTIONS, - EntityKey.METRICS_SUMMARIES, - EntityKey.REPLAYS, - EntityKey.__INTERNAL_ORG_METRICS_SETS, - EntityKey.TRANSACTIONS, - EntityKey.GENERIC_METRICS_SETS, - EntityKey.DISCOVER_TRANSACTIONS, - EntityKey.__INTERNAL_GENERIC_ORG_METRICS_DISTRIBUTIONS, - EntityKey.GENERIC_METRICS_COUNTERS, - EntityKey.__INTERNAL_ORG_METRICS_DISTRIBUTIONS, - EntityKey.GROUPASSIGNEE, EntityKey.ORG_METRICS_COUNTERS, EntityKey.__INTERNAL_GENERIC_ORG_METRICS_SETS, - EntityKey.ORG_SESSIONS, + EntityKey.DISCOVER, - EntityKey.GENERIC_METRICS_DISTRIBUTIONS, & ^^ ^^ + EntityKey.__INTERNAL_ORG_METRICS_DISTRIBUTIONS, & ^^^ + ^^^^^^^ + EntityKey.__INTERNAL_ORG_METRICS_SETS, - EntityKey.GENERIC_METRICS_GAUGES, - EntityKey.OUTCOMES, EntityKey.GROUPEDMESSAGE, - EntityKey.SESSIONS, - EntityKey.SEARCH_ISSUES, - EntityKey.OUTCOMES_RAW, }

[volokluev]

LGTM just some changes to allocation policy suggested and another test

[enochtangg]

LGTM. Not sure if you plan to do this in another PR, but you would also need to reference the new entities in the generic metrics dataset YAML.

[nikhars]

LGTM