Set debug level in system tests for agent

[mrodm]

Closes #1659

Update system tests runner to set log level "debug" during the tests. Once the test finishes a new handler has been added to restore the same log level it was set at the beginning during the tearDown process.

How to test locally

# start version 8.12.0
elastic-package stack up -v --version 8.12.0

# go to prometheus package in integrations repository and run system tests
cd /path/to/integrations/packages/prometheues

# Running system tests with Elastic stack 8.12.0 should raise errors
elastic-package test system -v

elastic-package stack down -v

elastic-package stack up -v -d --version 8.12.1
# no errors running system tests should be reported in 8.12.1
elastic-package test system -v
elastic-package stack down -v

Running test system should detect error related to log messages:

--- Test results for package: prometheus - START ---
FAILURE DETAILS:
prometheus/collector (elastic-agent logs):
[0] found error "Cannot index event publisher.Event...,\"reason\":\"mapper [prometheus.prometheus_http_response_size_bytes.histogram.values] cannot be changed from type [long] to [float]\"}, dropping event!"
[1] found error "Cannot index event publisher.Event....Cache:publisher.EventCache{m:mapstr.M(nil)}} (status=400): {\"type\":\"illegal_argument_exception\",\"reason\":\"mapper [prometheus.prometheus_tsdb_compaction_chunk_size_bytes.histogram.values] cannot be changed from type [long] to [float]\"}, dropping event!"


╭────────────┬─────────────┬───────────┬─────────────────────────────────┬────────────────────────────────────────────────────────────────────────────────────┬───────────────╮
│ PACKAGE    │ DATA STREAM │ TEST TYPE │ TEST NAME                       │ RESULT                                                                             │  TIME ELAPSED │
├────────────┼─────────────┼───────────┼─────────────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┼───────────────┤
│ prometheus │ collector   │ system    │ default (variant: prometheus_2) │ PASS                                                                               │ 30.136575255s │
│ prometheus │ collector   │ system    │ (elastic-agent logs)            │ FAIL: test case failed: one or more errors found while examining elastic-agent.log │  112.981574ms │
╰────────────┴─────────────┴───────────┴─────────────────────────────────┴────────────────────────────────────────────────────────────────────────────────────┴───────────────╯
--- Test results for package: prometheus - END   ---

Logs found

[0] found error "Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Date(2024, time.February, 6, 17, 31, 3, 721253526, time.Local), Meta:{\"input_id\":\"prometheus/metrics-prometheus-20181e88-74c8-4160-a37b-c6bc1cd8bbfa\",\"raw_index\":\"metrics-prometheus.collector-ep\",\"stream_id\":\"prometheus/metrics-prometheus.collector-20181e88-74c8-4160-a37b-c6bc1cd8bbfa\"}, Fields:{\"agent\":{\"ephemeral_id\":\"dbe96641-7e99-42f7-9767-acc02e3ad032\",\"id\":\"cabad701-9722-45af-b8fb-9972197967b4\",\"name\":\"docker-fleet-agent\",\"type\":\"metricbeat\",\"version\":\"8.12.0\"},\"data_stream\":{\"dataset\":\"prometheus.collector\",\"namespace\":\"ep\",\"type\":\"metrics\"},\"ecs\":{\"version\":\"8.0.0\"},\"elastic_agent\":{\"id\":\"cabad701-9722-45af-b8fb-9972197967b4\",\"snapshot\":false,\"version\":\"8.12.0\"},\"event\":{\"dataset\":\"prometheus.collector\",\"duration\":1971822677,\"module\":\"prometheus\"},\"host\":{\"architecture\":\"x86_64\",\"containerized\":false,\"hostname\":\"docker-fleet-agent\",\"id\":\"829324aac17946dcace17006fa82a2d2\",\"ip\":[\"192.168.176.7\"],\"mac\":[\"02-42-C0-A8-B0-07\"],\"name\":\"docker-fleet-agent\",\"os\":{\"codename\":\"focal\",\"family\":\"debian\",\"kernel\":\"6.5.0-15-generic\",\"name\":\"Ubuntu\",\"platform\":\"ubuntu\",\"type\":\"linux\",\"version\":\"20.04.6 LTS (Focal Fossa)\"}},\"metricset\":{\"name\":\"collector\",\"period\":10000},\"prometheus\":{\"labels\":{\"handler\":\"/metrics\",\"instance\":\"elastic-package-service-prometheus-1:9090\",\"job\":\"prometheus\"},\"prometheus_http_request_duration_seconds\":{\"histogram\":{\"counts\":[0,0,0,0,0,0,0,0,0,0],\"values\":[0.05,0.15000000000000002,0.30000000000000004,0.7,2,5.5,14,40,90,120]}},\"prometheus_http_response_size_bytes\":{\"histogram\":{\"counts\":[0,0,0,0,0,0,0,0,0],\"values\":[50,550,5500,55000,550000,5500000,55000000,550000000,1000000000]}}},\"service\":{\"address\":\"http://elastic-package-service-prometheus-1:9090/metrics\",\"type\":\"prometheus\"}}, Private:interface {}(nil), TimeSeries:true}, Flags:0x0, Cache:publisher.EventCache{m:mapstr.M(nil)}} (status=400): {\"type\":\"illegal_argument_exception\",\"reason\":\"mapper [prometheus.prometheus_http_response_size_bytes.histogram.values] cannot be changed from type [long] to [float]\"}, dropping event!"
[1] found error "Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Date(2024, time.February, 6, 17, 31, 3, 721253526, time.Local), Meta:{\"input_id\":\"prometheus/metrics-prometheus-20181e88-74c8-4160-a37b-c6bc1cd8bbfa\",\"raw_index\":\"metrics-prometheus.collector-ep\",\"stream_id\":\"prometheus/metrics-prometheus.collector-20181e88-74c8-4160-a37b-c6bc1cd8bbfa\"}, Fields:{\"agent\":{\"ephemeral_id\":\"dbe96641-7e99-42f7-9767-acc02e3ad032\",\"id\":\"cabad701-9722-45af-b8fb-9972197967b4\",\"name\":\"docker-fleet-agent\",\"type\":\"metricbeat\",\"version\":\"8.12.0\"},\"data_stream\":{\"dataset\":\"prometheus.collector\",\"namespace\":\"ep\",\"type\":\"metrics\"},\"ecs\":{\"version\":\"8.0.0\"},\"elastic_agent\":{\"id\":\"cabad701-9722-45af-b8fb-9972197967b4\",\"snapshot\":false,\"version\":\"8.12.0\"},\"event\":{\"dataset\":\"prometheus.collector\",\"duration\":1971908699,\"module\":\"prometheus\"},\"host\":{\"architecture\":\"x86_64\",\"containerized\":false,\"hostname\":\"docker-fleet-agent\",\"id\":\"829324aac17946dcace17006fa82a2d2\",\"ip\":[\"192.168.176.7\"],\"mac\":[\"02-42-C0-A8-B0-07\"],\"name\":\"docker-fleet-agent\",\"os\":{\"codename\":\"focal\",\"family\":\"debian\",\"kernel\":\"6.5.0-15-generic\",\"name\":\"Ubuntu\",\"platform\":\"ubuntu\",\"type\":\"linux\",\"version\":\"20.04.6 LTS (Focal Fossa)\"}},\"metricset\":{\"name\":\"collector\",\"period\":10000},\"prometheus\":{\"go_gc_duration_seconds_count\":{\"counter\":5,\"rate\":0},\"go_gc_duration_seconds_sum\":{\"counter\":0.00051995,\"rate\":0},\"go_goroutines\":{\"value\":31},\"go_memstats_alloc_bytes\":{\"value\":13497336},\"go_memstats_alloc_bytes_total\":{\"counter\":21237824,\"rate\":0},\"go_memstats_buck_hash_sys_bytes\":{\"value\":1458830},\"go_memstats_frees_total\":{\"counter\":53349,\"rate\":0},\"go_memstats_gc_sys_bytes\":{\"value\":5458128},\"go_memstats_heap_alloc_bytes\":{\"value\":13497336},\"go_memstats_heap_idle_bytes\":{\"value\":4521984},\"go_memstats_heap_inuse_bytes\":{\"value\":15040512},\"go_memstats_heap_objects\":{\"value\":84160},\"go_memstats_heap_released_bytes\":{\"value\":3989504},\"go_memstats_heap_sys_bytes\":{\"value\":19562496},\"go_memstats_last_gc_time_seconds\":{\"value\":1707240652.5910935},\"go_memstats_lookups_total\":{\"counter\":0,\"rate\":0},\"go_memstats_mallocs_total\":{\"counter\":137509,\"rate\":0},\"go_memstats_mcache_inuse_bytes\":{\"value\":19200},\"go_memstats_mcache_sys_bytes\":{\"value\":31200},\"go_memstats_mspan_inuse_bytes\":{\"value\":269416},\"go_memstats_mspan_sys_bytes\":{\"value\":277440},\"go_memstats_next_gc_bytes\":{\"value\":19836960},\"go_memstats_other_sys_bytes\":{\"value\":2556682},\"go_memstats_stack_inuse_bytes\":{\"value\":1409024},\"go_memstats_stack_sys_bytes\":{\"value\":1409024},\"go_memstats_sys_bytes\":{\"value\":30753800},\"go_threads\":{\"value\":21},\"labels\":{\"instance\":\"elastic-package-service-prometheus-1:9090\",\"job\":\"prometheus\"},\"process_cpu_seconds_total\":{\"counter\":0.06,\"rate\":0},\"process_max_fds\":{\"value\":1048576},\"process_open_fds\":{\"value\":14},\"process_resident_memory_bytes\":{\"value\":60293120},\"process_start_time_seconds\":{\"value\":1707240646.46},\"process_virtual_memory_bytes\":{\"value\":803663872},\"process_virtual_memory_max_bytes\":{\"value\":18446744073709552000},\"prometheus_api_remote_read_queries\":{\"value\":0},\"prometheus_config_last_reload_success_timestamp_seconds\":{\"value\":1707240647.5884995},\"prometheus_config_last_reload_successful\":{\"value\":1},\"prometheus_engine_queries\":{\"value\":0},\"prometheus_engine_queries_concurrent_max\":{\"value\":20},\"prometheus_engine_query_log_enabled\":{\"value\":0},\"prometheus_engine_query_log_failures_total\":{\"counter\":0,\"rate\":0},\"prometheus_notifications_alertmanagers_discovered\":{\"value\":0},\"prometheus_notifications_dropped_total\":{\"counter\":0,\"rate\":0},\"prometheus_notifications_queue_capacity\":{\"value\":10000},\"prometheus_notifications_queue_length\":{\"value\":0},\"prometheus_ready\":{\"value\":1},\"prometheus_remote_storage_exemplars_in_total\":{\"counter\":0,\"rate\":0},\"prometheus_remote_storage_highest_timestamp_in_seconds\":{\"value\":1707240660},\"prometheus_remote_storage_samples_in_total\":{\"counter\":345,\"rate\":0},\"prometheus_remote_storage_string_interner_zero_reference_releases_total\":{\"counter\":0,\"rate\":0},\"prometheus_rule_evaluation_duration_seconds_count\":{\"counter\":0,\"rate\":0},\"prometheus_rule_evaluation_duration_seconds_sum\":{\"counter\":0,\"rate\":0},\"prometheus_rule_group_duration_seconds_count\":{\"counter\":0,\"rate\":0},\"prometheus_rule_group_duration_seconds_sum\":{\"counter\":0,\"rate\":0},\"prometheus_sd_azure_failures_total\":{\"counter\":0,\"rate\":0},\"prometheus_sd_consul_rpc_failures_total\":{\"counter\":0,\"rate\":0},\"prometheus_sd_dns_lookup_failures_total\":{\"counter\":0,\"rate\":0},\"prometheus_sd_dns_lookups_total\":{\"counter\":0,\"rate\":0},\"prometheus_sd_file_read_errors_total\":{\"counter\":0,\"rate\":0},\"prometheus_sd_file_scan_duration_seconds_count\":{\"counter\":0,\"rate\":0},\"prometheus_sd_file_scan_duration_seconds_sum\":{\"counter\":0,\"rate\":0},\"prometheus_sd_http_failures_total\":{\"counter\":0,\"rate\":0},\"prometheus_sd_kuma_fetch_duration_seconds_count\":{\"counter\":0,\"rate\":0},\"prometheus_sd_kuma_fetch_duration_seconds_sum\":{\"counter\":0,\"rate\":0},\"prometheus_sd_kuma_fetch_failures_total\":{\"counter\":0,\"rate\":0},\"prometheus_sd_kuma_fetch_skipped_updates_total\":{\"counter\":0,\"rate\":0},\"prometheus_sd_linode_failures_total\":{\"counter\":0,\"rate\":0},\"prometheus_target_scrape_pool_exceeded_label_limits_total\":{\"counter\":0,\"rate\":0},\"prometheus_target_scrape_pool_exceeded_target_limit_total\":{\"counter\":0,\"rate\":0},\"prometheus_target_scrape_pool_reloads_failed_total\":{\"counter\":0,\"rate\":0},\"prometheus_target_scrape_pool_reloads_total\":{\"counter\":0,\"rate\":0},\"prometheus_target_scrape_pools_failed_total\":{\"counter\":0,\"rate\":0},\"prometheus_target_scrape_pools_total\":{\"counter\":1,\"rate\":0},\"prometheus_target_scrapes_cache_flush_forced_total\":{\"counter\":0,\"rate\":0},\"prometheus_target_scrapes_exceeded_body_size_limit_total\":{\"counter\":0,\"rate\":0},\"prometheus_target_scrapes_exceeded_sample_limit_total\":{\"counter\":0,\"rate\":0},\"prometheus_target_scrapes_exemplar_out_of_order_total\":{\"counter\":0,\"rate\":0},\"prometheus_target_scrapes_sample_duplicate_timestamp_total\":{\"counter\":0,\"rate\":0},\"prometheus_target_scrapes_sample_out_of_bounds_total\":{\"counter\":0,\"rate\":0},\"prometheus_target_scrapes_sample_out_of_order_total\":{\"counter\":0,\"rate\":0},\"prometheus_template_text_expansion_failures_total\":{\"counter\":0,\"rate\":0},\"prometheus_template_text_expansions_total\":{\"counter\":0,\"rate\":0},\"prometheus_treecache_watcher_goroutines\":{\"value\":0},\"prometheus_treecache_zookeeper_failures_total\":{\"counter\":0,\"rate\":0},\"prometheus_tsdb_blocks_loaded\":{\"value\":0},\"prometheus_tsdb_checkpoint_creations_failed_total\":{\"counter\":0,\"rate\":0},\"prometheus_tsdb_checkpoint_creations_total\":{\"counter\":0,\"rate\":0},\"prometheus_tsdb_checkpoint_deletions_failed_total\":{\"counter\":0,\"rate\":0},\"prometheus_tsdb_checkpoint_deletions_total\":{\"counter\":0,\"rate\":0},\"prometheus_tsdb_clean_start\":{\"value\":1},\"prometheus_tsdb_compaction_chunk_range_seconds\":{\"histogram\":{\"counts\":[0,0,0,0,0,0,0,0,0,0,0],\"values\":[50,250,1000,4000,16000,64000,256000,1024000,4096000,16384000,26214400]}},\"prometheus_tsdb_compaction_chunk_samples\":{\"histogram\":{\"counts\":[0,0,0,0,0,0,0,0,0,0,0,0,0],\"values\":[2,5,7.5,11.25,16.875,25.3125,37.96875,56.953125,85.4296875,128.14453125,192.216796875,288.3251953125,345.990234375]}},\"prometheus_tsdb_compaction_chunk_size_bytes\":{\"histogram\":{\"counts\":[0,0,0,0,0,0,0,0,0,0,0,0,0],\"values\":[16,40,60,90,135,202.5,303.75,455.625,683.4375,1025.15625,1537.734375,2306.6015625,2767.921875]}},\"prometheus_tsdb_compaction_duration_seconds\":{\"histogram\":{\"counts\":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],\"values\":[0.5,1.5,3,6,12,24,48,96,192,384,768,1536,3072,6144,8192]}},\"prometheus_tsdb_compaction_populating_block\":{\"value\":0},\"prometheus_tsdb_compactions_failed_total\":{\"counter\":0,\"rate\":0},\"prometheus_tsdb_compactions_skipped_total\":{\"counter\":0,\"rate\":0},\"prometheus_tsdb_compactions_total\":{\"counter\":0,\"rate\":0},\"prometheus_tsdb_compactions_triggered_total\":{\"counter\":0,\"rate\":0},\"prometheus_tsdb_data_replay_duration_seconds\":{\"value\":0.000151605},\"prometheus_tsdb_exemplar_exemplars_appended_total\":{\"counter\":0,\"rate\":0},\"prometheus_tsdb_exemplar_exemplars_in_storage\":{\"value\":0},\"prometheus_tsdb_exemplar_last_exemplars_timestamp_seconds\":{\"value\":0},\"prometheus_tsdb_exemplar_max_exemplars\":{\"value\":0},\"prometheus_tsdb_exemplar_out_of_order_exemplars_total\":{\"counter\":0,\"rate\":0},\"prometheus_tsdb_exemplar_series_with_exemplars_in_storage\":{\"value\":0},\"prometheus_tsdb_head_active_appenders\":{\"value\":0},\"prometheus_tsdb_head_chunks\":{\"value\":345},\"prometheus_tsdb_head_chunks_created_total\":{\"counter\":345,\"rate\":0},\"prometheus_tsdb_head_chunks_removed_total\":{\"counter\":0,\"rate\":0},\"prometheus_tsdb_head_gc_duration_seconds_count\":{\"counter\":0,\"rate\":0},\"prometheus_tsdb_head_gc_duration_seconds_sum\":{\"counter\":0,\"rate\":0},\"prometheus_tsdb_head_max_time\":{\"value\":1707240660702},\"prometheus_tsdb_head_max_time_seconds\":{\"value\":1707240660},\"prometheus_tsdb_head_min_time\":{\"value\":1707240660702},\"prometheus_tsdb_head_min_time_seconds\":{\"value\":1707240660},\"prometheus_tsdb_head_samples_appended_total\":{\"counter\":345,\"rate\":0},\"prometheus_tsdb_head_series\":{\"value\":345},\"prometheus_tsdb_head_series_created_total\":{\"counter\":345,\"rate\":0},\"prometheus_tsdb_head_series_not_found_total\":{\"counter\":0,\"rate\":0},\"prometheus_tsdb_head_series_removed_total\":{\"counter\":0,\"rate\":0},\"prometheus_tsdb_head_truncations_failed_total\":{\"counter\":0,\"rate\":0},\"prometheus_tsdb_head_truncations_total\":{\"counter\":0,\"rate\":0},\"prometheus_tsdb_isolation_high_watermark\":{\"value\":1},\"prometheus_tsdb_isolation_low_watermark\":{\"value\":1},\"prometheus_tsdb_lowest_timestamp\":{\"value\":1707240660702},\"prometheus_tsdb_lowest_timestamp_seconds\":{\"value\":1707240660},\"prometheus_tsdb_mmap_chunk_corruptions_total\":{\"counter\":0,\"rate\":0},\"prometheus_tsdb_out_of_bound_samples_total\":{\"counter\":0,\"rate\":0},\"prometheus_tsdb_out_of_order_samples_total\":{\"counter\":0,\"rate\":0},\"prometheus_tsdb_reloads_failures_total\":{\"counter\":0,\"rate\":0},\"prometheus_tsdb_reloads_total\":{\"counter\":1,\"rate\":0},\"prometheus_tsdb_retention_limit_bytes\":{\"value\":0},\"prometheus_tsdb_size_retentions_total\":{\"counter\":0,\"rate\":0},\"prometheus_tsdb_snapshot_replay_error_total\":{\"counter\":0,\"rate\":0},\"prometheus_tsdb_storage_blocks_bytes\":{\"value\":0},\"prometheus_tsdb_symbol_table_size_bytes\":{\"value\":0},\"prometheus_tsdb_time_retentions_total\":{\"counter\":0,\"rate\":0},\"prometheus_tsdb_tombstone_cleanup_seconds\":{\"histogram\":{\"counts\":[0,0,0,0,0,0,0,0,0,0,0,0],\"values\":[0.0025,0.0075,0.0175,0.037500000000000006,0.07500000000000001,0.175,0.375,0.75,1.75,3.75,7.5,10]}},\"prometheus_tsdb_vertical_compactions_total\":{\"counter\":0,\"rate\":0},\"prometheus_tsdb_wal_completed_pages_total\":{\"counter\":0,\"rate\":0},\"prometheus_tsdb_wal_corruptions_total\":{\"counter\":0,\"rate\":0},\"prometheus_tsdb_wal_fsync_duration_seconds_count\":{\"counter\":0,\"rate\":0},\"prometheus_tsdb_wal_fsync_duration_seconds_sum\":{\"counter\":0,\"rate\":0},\"prometheus_tsdb_wal_page_flushes_total\":{\"counter\":2,\"rate\":0},\"prometheus_tsdb_wal_segment_current\":{\"value\":0},\"prometheus_tsdb_wal_truncate_duration_seconds_count\":{\"counter\":0,\"rate\":0},\"prometheus_tsdb_wal_truncate_duration_seconds_sum\":{\"counter\":0,\"rate\":0},\"prometheus_tsdb_wal_truncations_failed_total\":{\"counter\":0,\"rate\":0},\"prometheus_tsdb_wal_truncations_total\":{\"counter\":0,\"rate\":0},\"prometheus_tsdb_wal_writes_failed_total\":{\"counter\":0,\"rate\":0},\"prometheus_web_federation_errors_total\":{\"counter\":0,\"rate\":0},\"prometheus_web_federation_warnings_total\":{\"counter\":0,\"rate\":0},\"promhttp_metric_handler_requests_in_flight\":{\"value\":1},\"up\":{\"value\":1}},\"service\":{\"address\":\"http://elastic-package-service-prometheus-1:9090/metrics\",\"type\":\"prometheus\"}}, Private:interface {}(nil), TimeSeries:true}, Flags:0x0, Cache:publisher.EventCache{m:mapstr.M(nil)}} (status=400): {\"type\":\"illegal_argument_exception\",\"reason\":\"mapper [prometheus.prometheus_tsdb_compaction_chunk_size_bytes.histogram.values] cannot be changed from type [long] to [float]\"}, dropping event!"

[ebeahan]

I'm in favor of making this change, even beyond the context discussed in #1659.

[jsoriano]

Thanks! 👍

[jsoriano]

Cool, I wouldn't have thought on actions to make this change 👍

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: eaaf518

History

• 💚 Build #2274 succeeded d877089

cc @mrodm