Skip to content

[CI] Add Buildkite step to run Python CI scripts unit tests#18942

Merged
mrodm merged 6 commits into
elastic:mainfrom
mrodm:add_tests_for_find_oldest_supported_version
May 12, 2026
Merged

[CI] Add Buildkite step to run Python CI scripts unit tests#18942
mrodm merged 6 commits into
elastic:mainfrom
mrodm:add_tests_for_find_oldest_supported_version

Conversation

@mrodm
Copy link
Copy Markdown
Collaborator

@mrodm mrodm commented May 11, 2026 •
edited
Loading

Proposed commit message

Add a generic Buildkite pipeline step (check-ci-python-scripts) that runs
unit tests for Python scripts used in CI.

NOTE: It is used the provider gcp in Buildkite to ensure that the same python3 version as in the integration steps is used.

WHAT:

  • New step :snake: Python CI scripts unit tests in pipeline.yml that runs
    run_ci_python_scripts_tests.sh on a GCP Ubuntu agent.
  • The step runs in parallel with the existing check (Go sources) step, keeping
    CI feedback fast.
  • test-integrations is gated on both check and check-ci-python-scripts
    passing before triggering package tests.
  • The shell script sets up an isolated Python venv, installs pinned deps from
    requirements-ci-python-scripts.txt, and invokes find_oldest_supported_version.py --test.
  • .gitignore updated to exclude the generated venv directory.
  • list_all_directories in common.sh temporarily filtered to
    packages/elastic_package_registry for focused testing (to be reverted).

WHY:
find_oldest_supported_version.py contains logic critical to how the CI
pipeline selects which stack versions to test against. Without a CI gate, a
regression in that script would silently affect all integration test runs.
This PR adds that safety net and lays the groundwork for testing other Python
CI scripts in the same step.

Author's Checklist

  • Revert the temporary grep filter in list_all_directories (common.sh) before merging.
  • Trigger a Buildkite build on this branch and verify the new :snake: Python CI scripts unit tests step appears and passes.
  • Confirm test-integrations only starts after both check and check-ci-python-scripts succeed.
  • Run .buildkite/scripts/run_ci_python_scripts_tests.sh locally to verify the venv setup and tests pass.

This PR was generated with the assistance of Claude (claude-sonnet-4-6).

mrodm and others added 2 commits May 11, 2026 19:06
@mrodm @claude
Add generic Buildkite step to run Python CI scripts unit tests
f563058 
Introduces a reusable pipeline step (`check-ci-python-scripts`) that runs
unit tests for Python scripts used in CI, starting with
`find_oldest_supported_version.py`. The step runs in an isolated venv and
in parallel with the Go source check. The `test-integrations` step is gated
on it passing.

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
@mrodm
Update listPackages temporarily - to be removed
d0339ca
@mrodm mrodm self-assigned this May 11, 2026
@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented May 11, 2026

TL;DR

The Buildkite failure is caused by virtualenv creation in the new :snake: Python CI scripts unit tests step, not by a test assertion. The step runs on an Ubuntu agent image without python3-venv/ensurepip, so python3 -m venv fails and blocks the rest of the pipeline.

Remediation

  • Make venv creation deterministic on CI agents: either run this step on an image that already includes python3-venv, or install python3-venv unconditionally before creating the venv (instead of only when id -u == 0).
  • Update .buildkite/scripts/run_ci_python_scripts_tests.sh to fail fast with a clear error when python3 -m venv is unavailable after setup.
  • Re-run Buildkite build 42665 (or a rebuild of the same PR commit) and confirm :snake: Python CI scripts unit tests passes.
Investigation details

Root Cause

run_ci_python_scripts_tests.sh creates a venv with python3 -m venv and retries after optional apt install. However, the apt install path is gated to root only:

  • .buildkite/scripts/run_ci_python_scripts_tests.sh line 14: python3 -m venv "${VENV_DIR}"
  • line 15: if command -v apt-get ... && [[ "$(id -u)" -eq 0 ]]; then
  • line 21: retries python3 -m venv "${VENV_DIR}"

On this Buildkite worker, ensurepip is missing, and because the install branch is conditional on root, the second venv attempt fails the same way.

The step was introduced in this PR and runs on Ubuntu image family:

  • .buildkite/pipeline.yml lines 63-69 (check-ci-python-scripts, image: "${IMAGE_UBUNTU_X86_64}").

Evidence

The virtual environment was not created successfully because ensurepip is not available.
... apt install python3.10-venv ...
Failing command: .../.buildkite/.venv-ci-python-scripts/bin/python3

Verification

  • Not run locally; analysis is based on Buildkite logs and PR branch script/pipeline changes.

Follow-up

  • If you prefer not to manage apt packages in-step, pin this step to an agent image that already provides python3-venv and keep the script strictly test-focused.

Note

🔒 Integrity filter blocked 2 items

The following items were blocked because they don't meet the GitHub integrity level.

To allow these resources, lower min-integrity in your GitHub frontmatter:

tools:
  github:
    min-integrity: approved  # merged | approved | unapproved | none

What is this? | From workflow: PR Buildkite Detective

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.

mrodm and others added 2 commits May 11, 2026 19:24
@mrodm @claude
Fix python3 venv setup and pip invocation in CI Python scripts test r…
ded4735 
…unner

- Replace bare `pip`/`python` calls with `python3 -m pip` and `python3`
  to ensure the venv-scoped interpreter is used explicitly.
- Drop the root check for the python3-venv fallback install and use `sudo`
  following the same pattern as `with_docker` in common.sh, fixing a CI
  failure where the Buildkite agent runs as a non-root user.

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
@mrodm
Add collapsible sections in Buildkite output
770f0ae
@elastic-vault-github-plugin-prod
Copy link
Copy Markdown

elastic-vault-github-plugin-prod Bot commented May 11, 2026 •
edited
Loading

🚀 Benchmarks report

Package abnormal_security 👍(3) 💚(1) 💔(2)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
threat 3448.28 2793.3 -654.98 (-18.99%) 💔
vendor_case 12195.12 8333.33 -3861.79 (-31.67%) 💔

Package admin_by_request_epm 👍(1) 💚(0) 💔(1)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
events 15873.02 10416.67 -5456.35 (-34.37%) 💔

Package agentless_hello_world 👍(0) 💚(1) 💔(1)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
generic 55555.56 43478.26 -12077.3 (-21.74%) 💔

Package airlock_digital 👍(2) 💚(0) 💔(1)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
server_activities 20833.33 14492.75 -6340.58 (-30.43%) 💔

Package apache_tomcat 👍(2) 💚(3) 💔(4)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
cache 25000 16949.15 -8050.85 (-32.2%) 💔
catalina 12345.68 9708.74 -2636.94 (-21.36%) 💔
localhost 23255.81 15625 -7630.81 (-32.81%) 💔
memory 22222.22 18518.52 -3703.7 (-16.67%) 💔

Package armis 👍(1) 💚(0) 💔(2)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
alert 2638.52 2127.66 -510.86 (-19.36%) 💔
device 2123.14 1543.21 -579.93 (-27.31%) 💔

Package auditd 👍(0) 💚(0) 💔(1)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
log 2036.66 1605.14 -431.52 (-21.19%) 💔

Package aws_bedrock_agentcore 👍(1) 💚(1) 💔(1)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
gateway_application_logs 5263.16 3225.81 -2037.35 (-38.71%) 💔

Package awsfirehose 👍(1) 💚(0) 💔(1)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
metrics 5319.15 4366.81 -952.34 (-17.9%) 💔

Package azure 👍(5) 💚(2) 💔(5)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
provisioning 13157.89 8064.52 -5093.37 (-38.71%) 💔
activitylogs 2217.29 1848.43 -368.86 (-16.64%) 💔
application_gateway 6369.43 5025.13 -1344.3 (-21.11%) 💔
auditlogs 5181.35 3039.51 -2141.84 (-41.34%) 💔
identity_protection 5780.35 3246.75 -2533.6 (-43.83%) 💔

Package azure_metrics 👍(1) 💚(1) 💔(2)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
container_instance 500000 333333.33 -166666.67 (-33.33%) 💔
container_service 333333.33 250000 -83333.33 (-25%) 💔

Package barracuda_cloudgen_firewall 👍(0) 💚(0) 💔(1)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
log 8547.01 7246.38 -1300.63 (-15.22%) 💔

Package beyondinsight_password_safe 👍(2) 💚(1) 💔(2)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
session 7518.8 6211.18 -1307.62 (-17.39%) 💔
useraudit 10204.08 7633.59 -2570.49 (-25.19%) 💔

Package bitdefender 👍(1) 💚(1) 💔(1)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
push_statistics 83333.33 55555.56 -27777.77 (-33.33%) 💔

Package blacklens 👍(0) 💚(0) 💔(1)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
alerts 7518.8 4504.5 -3014.3 (-40.09%) 💔

Package carbon_black_cloud 👍(1) 💚(1) 💔(3)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
asset_vulnerability_summary 15151.52 12048.19 -3103.33 (-20.48%) 💔
audit 18867.92 13513.51 -5354.41 (-28.38%) 💔
endpoint_event 1821.49 1422.48 -399.01 (-21.91%) 💔

Package cassandra 👍(0) 💚(0) 💔(1)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
log 5524.86 4524.89 -999.97 (-18.1%) 💔

Package ceph 👍(3) 💚(2) 💔(2)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
cluster_status 4201.68 3389.83 -811.85 (-19.32%) 💔
osd_tree 25641.03 15625 -10016.03 (-39.06%) 💔

Package checkpoint_email 👍(0) 💚(0) 💔(1)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
event 1930.5 1594.9 -335.6 (-17.38%) 💔

Package checkpoint_harmony_endpoint 👍(5) 💚(1) 💔(1)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
zerophishing 8849.56 4878.05 -3971.51 (-44.88%) 💔

Package cisco_duo 👍(0) 💚(5) 💔(3)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
admin 6802.72 5050.51 -1752.21 (-25.76%) 💔
auth 3058.1 2247.19 -810.91 (-26.52%) 💔
summary 71428.57 47619.05 -23809.52 (-33.33%) 💔

Package cisco_nexus 👍(0) 💚(0) 💔(1)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
log 1908.4 1600 -308.4 (-16.16%) 💔

Package cisco_secure_email_gateway 👍(0) 💚(0) 💔(1)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
log 1883.24 1506.02 -377.22 (-20.03%) 💔

Package citrix_adc 👍(2) 💚(2) 💔(2)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
system 6097.56 4739.34 -1358.22 (-22.27%) 💔
vpn 6666.67 5434.78 -1231.89 (-18.48%) 💔

Package claroty_ctd 👍(1) 💚(1) 💔(1)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
baseline 3546.1 2304.15 -1241.95 (-35.02%) 💔

Package claroty_xdome 👍(1) 💚(1) 💔(1)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
alert 992.06 836.12 -155.94 (-15.72%) 💔

Package cloudflare_logpush 👍(19) 💚(11) 💔(12)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
gateway_dns 5347.59 4310.34 -1037.25 (-19.4%) 💔
sinkhole_http 7692.31 5405.41 -2286.9 (-29.73%) 💔
access_request 3891.05 3125 -766.05 (-19.69%) 💔
workers_trace 14084.51 8849.56 -5234.95 (-37.17%) 💔
audit 10000 4739.34 -5260.66 (-52.61%) 💔
http_request 1742.16 1406.47 -335.69 (-19.27%) 💔
nel_report 32258.06 25641.03 -6617.03 (-20.51%) 💔
spectrum_event 4524.89 3773.58 -751.31 (-16.6%) 💔
device_posture 7042.25 5291.01 -1751.24 (-24.87%) 💔
dlp_forensic_copies 21276.6 17543.86 -3732.74 (-17.54%) 💔
dns_firewall 7299.27 6060.61 -1238.66 (-16.97%) 💔
firewall_event 3773.58 2915.45 -858.13 (-22.74%) 💔

Package couchbase 👍(4) 💚(4) 💔(2)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
database_stats 31250 17543.86 -13706.14 (-43.86%) 💔
miscellaneous 13513.51 7575.76 -5937.75 (-43.94%) 💔

Package crowdstrike 👍(6) 💚(1) 💔(3)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
alert 1901.14 1600.77 -300.37 (-15.8%) 💔
falcon 7874.02 3259.45 -4614.57 (-58.61%) 💔
host 4484.3 3306.88 -1177.42 (-26.26%) 💔

Package cyberark_epm 👍(3) 💚(1) 💔(1)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
policyaudit_raw_event 3067.48 2450.98 -616.5 (-20.1%) 💔

Package cyberark_pta 👍(1) 💚(0) 💔(1)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
events 13698.63 7485.03 -6213.6 (-45.36%) 💔

Package cyberarkpas 👍(1) 💚(0) 💔(1)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
audit 1663.89 1326.26 -337.63 (-20.29%) 💔

Package cyera 👍(1) 💚(0) 💔(3)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
classification 6896.55 5649.72 -1246.83 (-18.08%) 💔
event 9615.38 8064.52 -1550.86 (-16.13%) 💔
issue 5714.29 4219.41 -1494.88 (-26.16%) 💔

Package cylance 👍(0) 💚(0) 💔(1)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
protect 58823.53 47619.05 -11204.48 (-19.05%) 💔

Package elasticsearch 👍(3) 💚(1) 💔(2)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
deprecation 7092.2 4587.16 -2505.04 (-35.32%) 💔
server 6250 4484.3 -1765.7 (-28.25%) 💔

Package entro 👍(0) 💚(0) 💔(1)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
audit 27027.03 20833.33 -6193.7 (-22.92%) 💔

Package envoyproxy 👍(0) 💚(0) 💔(1)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
log 1204.82 608.27 -596.55 (-49.51%) 💔

Package f5_bigip 👍(0) 💚(0) 💔(1)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
log 654.45 545.85 -108.6 (-16.59%) 💔

Package fireeye 👍(0) 💚(0) 💔(1)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
nx 1996.01 1602.56 -393.45 (-19.71%) 💔

Package first_epss 👍(0) 💚(0) 💔(1)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
vulnerability 52631.58 43478.26 -9153.32 (-17.39%) 💔

Package forcepoint_web 👍(0) 💚(0) 💔(1)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
logs 2500 2079 -421 (-16.84%) 💔

Package forgerock 👍(3) 💚(2) 💔(6)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
idm_core 71428.57 52631.58 -18796.99 (-26.32%) 💔
am_access 5882.35 4405.29 -1477.06 (-25.11%) 💔
am_activity 22727.27 19230.77 -3496.5 (-15.38%) 💔
am_config 52631.58 27777.78 -24853.8 (-47.22%) 💔
idm_access 23809.52 17241.38 -6568.14 (-27.59%) 💔
idm_authentication 55555.56 41666.67 -13888.89 (-25%) 💔

Package fortinet_fortigate 👍(0) 💚(0) 💔(1)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
log 981.35 772.2 -209.15 (-21.31%) 💔

Package fortinet_fortiproxy 👍(0) 💚(0) 💔(1)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
log 1245.33 914.08 -331.25 (-26.6%) 💔

Package gcp 👍(1) 💚(2) 💔(4)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
audit 1605.14 1104.97 -500.17 (-31.16%) 💔
compute 38461.54 32258.06 -6203.48 (-16.13%) 💔
dns 3717.47 2857.14 -860.33 (-23.14%) 💔
loadbalancing_logs 5494.51 3703.7 -1790.81 (-32.59%) 💔

Package github 👍(4) 💚(0) 💔(2)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
dependabot 5555.56 3690.04 -1865.52 (-33.58%) 💔
security_advisories 8849.56 6578.95 -2270.61 (-25.66%) 💔

Package gitlab 👍(5) 💚(0) 💔(2)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
application 2898.55 2242.15 -656.4 (-22.65%) 💔
audit 5747.13 4424.78 -1322.35 (-23.01%) 💔

Package google_scc 👍(2) 💚(1) 💔(1)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
source 45454.55 27027.03 -18427.52 (-40.54%) 💔

Package google_workspace 👍(14) 💚(2) 💔(6)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
drive 7092.2 5524.86 -1567.34 (-22.1%) 💔
keep 13333.33 10989.01 -2344.32 (-17.58%) 💔
login 9433.96 6756.76 -2677.2 (-28.38%) 💔
access_transparency 2881.84 2141.33 -740.51 (-25.7%) 💔
token 4000 2688.17 -1311.83 (-32.8%) 💔
data_studio 7352.94 6211.18 -1141.76 (-15.53%) 💔

Package grafana 👍(1) 💚(0) 💔(1)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
logs 9174.31 6289.31 -2885 (-31.45%) 💔

Package hadoop 👍(0) 💚(0) 💔(5)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
application 3846.15 2688.17 -1157.98 (-30.11%) 💔
cluster 6250 5076.14 -1173.86 (-18.78%) 💔
datanode 14492.75 10869.57 -3623.18 (-25%) 💔
namenode 12345.68 10309.28 -2036.4 (-16.49%) 💔
node_manager 24390.24 19607.84 -4782.4 (-19.61%) 💔

Package iis 👍(1) 💚(0) 💔(1)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
access 1336.9 1096.49 -240.41 (-17.98%) 💔

Package imperva 👍(0) 💚(0) 💔(1)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
securesphere 2777.78 1149.43 -1628.35 (-58.62%) 💔

Package island_browser 👍(3) 💚(0) 💔(2)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
audit 2785.52 1386.96 -1398.56 (-50.21%) 💔
device 3584.23 1669.45 -1914.78 (-53.42%) 💔

Package jamf_pro 👍(0) 💚(0) 💔(2)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
events 6451.61 3184.71 -3266.9 (-50.64%) 💔
inventory 6711.41 4545.45 -2165.96 (-32.27%) 💔

Package juniper_junos 👍(0) 💚(0) 💔(1)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
log 71428.57 58823.53 -12605.04 (-17.65%) 💔

Package juniper_netscreen 👍(0) 💚(0) 💔(1)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
log 41666.67 32258.06 -9408.61 (-22.58%) 💔

Package jupiter_one 👍(0) 💚(0) 💔(1)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
asset 1510.57 1088.14 -422.43 (-27.96%) 💔

Package lastpass 👍(0) 💚(1) 💔(2)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
detailed_shared_folder 9523.81 6493.51 -3030.3 (-31.82%) 💔
user 6666.67 4219.41 -2447.26 (-36.71%) 💔

Package lumos 👍(0) 💚(0) 💔(1)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
activity_logs 21739.13 14084.51 -7654.62 (-35.21%) 💔

Package m365_defender 👍(5) 💚(1) 💔(2)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
incident 1221 881.68 -339.32 (-27.79%) 💔
vulnerability 13157.89 8481.76 -4676.13 (-35.54%) 💔

Package macos 👍(2) 💚(3) 💔(2)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
system_change 7142.86 3703.7 -3439.16 (-48.15%) 💔
user_and_account_management 7575.76 3717.47 -3858.29 (-50.93%) 💔

Package menlo 👍(0) 💚(0) 💔(2)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
dlp 3039.51 2415.46 -624.05 (-20.53%) 💔
web 4065.04 3246.75 -818.29 (-20.13%) 💔

Package microsoft_defender_endpoint 👍(0) 💚(2) 💔(2)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
machine_action 6024.1 4926.11 -1097.99 (-18.23%) 💔
vulnerability 4405.29 3448.28 -957.01 (-21.72%) 💔

Package mimecast 👍(5) 💚(1) 💔(5)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
ttp_url_logs 15873.02 10309.28 -5563.74 (-35.05%) 💔
archive_search_logs 7462.69 6097.56 -1365.13 (-18.29%) 💔
dlp_logs 13333.33 7352.94 -5980.39 (-44.85%) 💔
siem_logs 3937.01 2439.02 -1497.99 (-38.05%) 💔
threat_intel_malware_grid 6289.31 3424.66 -2864.65 (-45.55%) 💔

Package miniflux 👍(0) 💚(0) 💔(1)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
feed_entry 2096.44 1572.33 -524.11 (-25%) 💔

Package mongodb 👍(0) 💚(0) 💔(1)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
log 29411.76 19230.77 -10180.99 (-34.62%) 💔

Package mongodb_atlas 👍(2) 💚(3) 💔(3)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
alert 2824.86 2164.5 -660.36 (-23.38%) 💔
disk 14492.75 8474.58 -6018.17 (-41.53%) 💔
hardware 16949.15 10989.01 -5960.14 (-35.16%) 💔

Package mysql 👍(1) 💚(1) 💔(1)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
error 33333.33 27777.78 -5555.55 (-16.67%) 💔

Package nagios_xi 👍(1) 💚(1) 💔(1)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
events 17857.14 13333.33 -4523.81 (-25.33%) 💔

Package neon_cyber 👍(1) 💚(0) 💔(1)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
events 3831.42 2967.36 -864.06 (-22.55%) 💔

Package netscout 👍(0) 💚(0) 💔(1)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
sightline 83333.33 62500 -20833.33 (-25%) 💔

Package netskope 👍(1) 💚(2) 💔(1)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
alerts 1582.28 1336.9 -245.38 (-15.51%) 💔

Package nginx_ingress_controller 👍(1) 💚(0) 💔(1)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
error 32258.06 20408.16 -11849.9 (-36.73%) 💔

Package nozomi_networks 👍(3) 💚(2) 💔(3)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
asset 2132.2 1597.44 -534.76 (-25.08%) 💔
node_cve 5208.33 3472.22 -1736.11 (-33.33%) 💔
session 11363.64 7462.69 -3900.95 (-34.33%) 💔

Package o365 👍(0) 💚(0) 💔(2)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
audit 2257.34 1440.92 -816.42 (-36.17%) 💔
audit 2257.34 1838.24 -419.1 (-18.57%) 💔

Package o365_metrics 👍(9) 💚(7) 💔(5)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
service_health 62500 52631.58 -9868.42 (-15.79%) 💔
teams_user_activity_user_detail 1757.47 1250 -507.47 (-28.88%) 💔
mailbox_usage_detail 3906.25 3205.13 -701.12 (-17.95%) 💔
mailbox_usage_quota_status 7352.94 5102.04 -2250.9 (-30.61%) 💔
onedrive_usage_file_counts 17857.14 10869.57 -6987.57 (-39.13%) 💔

Package oracle_weblogic 👍(0) 💚(2) 💔(2)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
domain 20833.33 12658.23 -8175.1 (-39.24%) 💔
managed_server 25641.03 15384.62 -10256.41 (-40%) 💔

Package panw_cortex_xdr 👍(1) 💚(1) 💔(1)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
alerts 946.97 718.91 -228.06 (-24.08%) 💔

Package panw_metrics 👍(0) 💚(0) 💔(1)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
system 16949.15 12658.23 -4290.92 (-25.32%) 💔

Package pfsense 👍(0) 💚(0) 💔(1)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
log 2873.56 2164.5 -709.06 (-24.68%) 💔

Package php_fpm 👍(0) 💚(0) 💔(2)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
pool 9174.31 6578.95 -2595.36 (-28.29%) 💔
process 8264.46 6024.1 -2240.36 (-27.11%) 💔

Package ping_one 👍(0) 💚(0) 💔(1)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
audit 2832.86 2369.67 -463.19 (-16.35%) 💔

Package platform_observability 👍(0) 💚(0) 💔(2)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
kibana_audit 9009.01 7352.94 -1656.07 (-18.38%) 💔
kibana_log 14925.37 10752.69 -4172.68 (-27.96%) 💔

Package prisma_cloud 👍(5) 💚(1) 💔(1)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
misconfiguration 9090.91 4926.11 -4164.8 (-45.81%) 💔

Package proofpoint_on_demand 👍(1) 💚(1) 💔(1)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
mail 4291.85 2832.86 -1458.99 (-33.99%) 💔

Package proxysg 👍(0) 💚(0) 💔(1)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
log 4184.1 2493.77 -1690.33 (-40.4%) 💔

Package qnap_nas 👍(0) 💚(0) 💔(1)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
log 2770.08 2252.25 -517.83 (-18.69%) 💔

Package redis 👍(0) 💚(0) 💔(1)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
log 9009.01 5405.41 -3603.6 (-40%) 💔

Package rubrik 👍(7) 💚(3) 💔(2)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
global_cluster_performance 11111.11 9174.31 -1936.8 (-17.43%) 💔
node_statistics 16666.67 11764.71 -4901.96 (-29.41%) 💔

Package salesforce 👍(1) 💚(1) 💔(2)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
logout 29411.76 8333.33 -21078.43 (-71.67%) 💔
setupaudittrail 9090.91 6060.61 -3030.3 (-33.33%) 💔

Package sentinel_one 👍(9) 💚(3) 💔(6)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
agent 3769.32 2949.85 -819.47 (-21.74%) 💔
application_risk 13422.82 8130.08 -5292.74 (-39.43%) 💔
group 26041.67 20000 -6041.67 (-23.2%) 💔
threat 1716.74 1410.44 -306.3 (-17.84%) 💔
threat_event 9372.07 4672.9 -4699.17 (-50.14%) 💔
application_risk 13422.82 9551.1 -3871.72 (-28.84%) 💔

Package sentinel_one_cloud_funnel 👍(0) 💚(0) 💔(1)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
event 378.93 313.68 -65.25 (-17.22%) 💔

Package servicenow 👍(0) 💚(0) 💔(1)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
event 4807.69 3174.6 -1633.09 (-33.97%) 💔

Package snyk 👍(0) 💚(1) 💔(1)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
issues 4115.23 3194.89 -920.34 (-22.36%) 💔

Package spring_boot 👍(2) 💚(0) 💔(1)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
gc 7751.94 5102.04 -2649.9 (-34.18%) 💔

Package spycloud 👍(1) 💚(1) 💔(1)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
breach_catalog 1182.03 955.11 -226.92 (-19.2%) 💔

Package stormshield 👍(0) 💚(0) 💔(1)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
log 2444.99 1953.13 -491.86 (-20.12%) 💔

Package symantec_endpoint_security 👍(0) 💚(1) 💔(1)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
event 148.96 102.75 -46.21 (-31.02%) 💔

Package syslog_router 👍(0) 💚(0) 💔(1)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
log 71428.57 18181.82 -53246.75 (-74.55%) 💔

Package system 👍(1) 💚(1) 💔(1)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
syslog 333333.33 166666.67 -166666.66 (-50%) 💔

Package tanium 👍(3) 💚(1) 💔(2)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
discover 5102.04 4081.63 -1020.41 (-20%) 💔
threat_response 775.19 612.75 -162.44 (-20.95%) 💔

Package teleport 👍(0) 💚(0) 💔(1)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
audit 590.32 394.01 -196.31 (-33.25%) 💔

Package tencent_cloud 👍(3) 💚(0) 💔(1)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
clb 3484.32 2531.65 -952.67 (-27.34%) 💔

Package ti_abusech 👍(4) 💚(4) 💔(4)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
ja3_fingerprints 17857.14 8298.76 -9558.38 (-53.53%) 💔
malware 19607.84 8904.72 -10703.12 (-54.59%) 💔
malwarebazaar 11494.25 8326.39 -3167.86 (-27.56%) 💔
threatfox 11668.61 9765.63 -1902.98 (-16.31%) 💔

Package ti_cyware_intel_exchange 👍(0) 💚(0) 💔(1)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
indicator 1923.08 1285.35 -637.73 (-33.16%) 💔

Package ti_domaintools 👍(1) 💚(2) 💔(3)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
domainhotlist_feed 55555.56 27777.78 -27777.78 (-50%) 💔
nad_feed 111111.11 71428.57 -39682.54 (-35.71%) 💔
nod_feed 62500 31250 -31250 (-50%) 💔

Package ti_eset 👍(3) 💚(1) 💔(3)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
botnet 15151.52 9433.96 -5717.56 (-37.74%) 💔
cc 21276.6 10638.3 -10638.3 (-50%) 💔
files 23255.81 18518.52 -4737.29 (-20.37%) 💔

Package ti_flashpoint 👍(2) 💚(0) 💔(1)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
alert 3105.59 2207.51 -898.08 (-28.92%) 💔

Package ti_google_threat_intelligence 👍(8) 💚(3) 💔(4)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
first_stage_delivery_vectors 4545.45 3773.58 -771.87 (-16.98%) 💔
iot 5882.35 4504.5 -1377.85 (-23.42%) 💔
malware 5617.98 4484.3 -1133.68 (-20.18%) 💔
mobile 6578.95 5319.15 -1259.8 (-19.15%) 💔

Package ti_maltiverse 👍(0) 💚(0) 💔(1)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
indicator 2666.67 1795.33 -871.34 (-32.68%) 💔

Package ti_misp 👍(1) 💚(0) 💔(1)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
threat_attributes 2994.01 2518.89 -475.12 (-15.87%) 💔

Package ti_recordedfuture 👍(1) 💚(0) 💔(2)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
threat 2232.14 1811.59 -420.55 (-18.84%) 💔
triggered_alert 5847.95 4878.05 -969.9 (-16.59%) 💔

Package ti_threatconnect 👍(0) 💚(0) 💔(1)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
indicator 529.38 424.63 -104.75 (-19.79%) 💔

Package tines 👍(0) 💚(1) 💔(1)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
audit_logs 3597.12 2409.64 -1187.48 (-33.01%) 💔

Package trellix_epo_cloud 👍(0) 💚(1) 💔(2)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
device 2288.33 1923.08 -365.25 (-15.96%) 💔
group 20408.16 14705.88 -5702.28 (-27.94%) 💔

Package trend_micro_vision_one 👍(5) 💚(5) 💔(2)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
network_activity 4472.27 3344.48 -1127.79 (-25.22%) 💔
detection 2590.67 2145.92 -444.75 (-17.17%) 💔

Package tychon 👍(11) 💚(0) 💔(8)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
features 3389.83 2341.92 -1047.91 (-30.91%) 💔
host 1677.85 1418.44 -259.41 (-15.46%) 💔
stig 2583.98 2092.05 -491.93 (-19.04%) 💔
systemcerts 1091.7 853.24 -238.46 (-21.84%) 💔
browser 1862.2 1000 -862.2 (-46.3%) 💔
ciphers 953.29 648.93 -304.36 (-31.93%) 💔
coams 2557.54 1976.28 -581.26 (-22.73%) 💔
cve 1547.99 930.23 -617.76 (-39.91%) 💔

Package vectra_rux 👍(1) 💚(2) 💔(2)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
audit 2881.84 1964.64 -917.2 (-31.83%) 💔
lockdown 15151.52 10869.57 -4281.95 (-28.26%) 💔

Package websphere_application_server 👍(1) 💚(1) 💔(2)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
jdbc 11235.96 9174.31 -2061.65 (-18.35%) 💔
servlet 21276.6 11235.96 -10040.64 (-47.19%) 💔

Package windows 👍(8) 💚(0) 💔(2)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
powershell_operational 2597.4 1475.8 -1121.6 (-43.18%) 💔
sysmon_operational 3030.3 2247.19 -783.11 (-25.84%) 💔

Package withsecure_elements 👍(1) 💚(0) 💔(1)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
security_events 10752.69 7812.5 -2940.19 (-27.34%) 💔

Package wiz 👍(3) 💚(1) 💔(2)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
audit 4424.78 2808.99 -1615.79 (-36.52%) 💔
cloud_configuration_finding 9090.91 4672.9 -4418.01 (-48.6%) 💔

Package zeek 👍(21) 💚(16) 💔(6)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
radius 41666.67 27027.03 -14639.64 (-35.14%) 💔
snmp 37037.04 31250 -5787.04 (-15.63%) 💔
ssh 38461.54 18867.92 -19593.62 (-50.94%) 💔
x509 23255.81 13698.63 -9557.18 (-41.1%) 💔
dhcp 28571.43 17543.86 -11027.57 (-38.6%) 💔
dpd 33333.33 22727.27 -10606.06 (-31.82%) 💔

Package zscaler_zia 👍(2) 💚(4) 💔(2)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
dns 1858.74 1538.46 -320.28 (-17.23%) 💔
firewall 1883.24 1388.89 -494.35 (-26.25%) 💔

Package zscaler_zpa 👍(4) 💚(0) 💔(1)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
user_status 5649.72 3816.79 -1832.93 (-32.44%) 💔

To see the full report comment with /test benchmark fullreport

@mrodm mrodm marked this pull request as ready for review May 11, 2026 17:56
@mrodm mrodm requested a review from a team as a code owner May 11, 2026 17:56
@elasticmachine
Copy link
Copy Markdown

elasticmachine commented May 11, 2026 •
edited
Loading

💔 Build Failed

Failed CI Steps

History

cc @mrodm

mrodm
mrodm commented May 11, 2026
View reviewed changes
Comment thread .buildkite/scripts/run_ci_python_scripts_tests.sh
REPO_ROOT="$(git rev-parse --show-toplevel)"
VENV_DIR="${REPO_ROOT}/.buildkite/.venv-ci-python-scripts"
REQ_FILE="${REPO_ROOT}/.buildkite/scripts/requirements-ci-python-scripts.txt"
FIND_OLDEST_SCRIPT="${REPO_ROOT}/.buildkite/scripts/find_oldest_supported_version.py"
Copy link
Copy Markdown
Collaborator Author

@mrodm mrodm May 11, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To execute these tests:

integrations/.buildkite/scripts/find_oldest_supported_version.py

Lines 126 to 274 in 22fb884

# Test: meant to run locally with --test
class TestFindOldestSupportVersion(unittest.TestCase):
"""Testcase for find_oldest_supported_version."""
mock_data = {
"versions": [
"7.17.10",
"7.17.11",
"7.17.12",
"7.17.13-SNAPSHOT",
"7.17.13",
"7.17.14-SNAPSHOT",
"8.7.0",
"8.7.1",
"8.8.0",
"8.8.1",
"8.8.2",
"8.9.1-SNAPSHOT",
"8.9.1",
"8.9.2-SNAPSHOT",
"8.9.2",
"8.9.4",
"8.10.0-SNAPSHOT",
"8.10.0",
"8.10.1-SNAPSHOT",
"8.10.1",
"8.11.0-SNAPSHOT"
],
"aliases": [
"7.x-SNAPSHOT",
"7.17-SNAPSHOT",
"7.17",
"8.x-SNAPSHOT",
"8.7",
"8.8",
"8.9-SNAPSHOT",
"8.9",
"8.10-SNAPSHOT",
"8.10",
"8.11-SNAPSHOT"
],
"manifests": {
"last-update-time": "Thu, 14 Sep 2023 16:03:46 UTC",
"seconds-since-last-update": 107
}
}
def setUp(self):
super().setUp()
global fetch_version
self._fetch_version = fetch_version
def fetch_version(): return self.mock_data
def tearDown(self):
global fetch_version
fetch_version = self._fetch_version
super().tearDown()
def test_next_patch_does_not_exits_and_available_version_contains_snapshot(self):
self.assertEqual(find_oldest_supported_version("7.17.14"), "7.17.14-SNAPSHOT")
self.assertEqual(find_oldest_supported_version("8.9.2"), "8.9.2-SNAPSHOT")
self.assertEqual(find_oldest_supported_version("8.10.1"), "8.10.1-SNAPSHOT")
self.assertEqual(find_oldest_supported_version("8.11.0"), "8.11.0-SNAPSHOT")
def test_available_version_contains_kibana_version(self):
self.assertEqual(find_oldest_supported_version("7.17.10"), "7.17.10")
self.assertEqual(find_oldest_supported_version("7.17.11"), "7.17.11")
self.assertEqual(find_oldest_supported_version("7.17.12"), "7.17.12")
self.assertEqual(find_oldest_supported_version("7.17.13"), "7.17.13")
self.assertEqual(find_oldest_supported_version("8.7.1"), "8.7.1")
self.assertEqual(find_oldest_supported_version("8.10.0"), "8.10.0")
def test_too_old_to_be_in_api(self):
self.assertEqual(find_oldest_supported_version("7.16.0"), "7.16.0")
self.assertEqual(find_oldest_supported_version("8.6.0"), "8.6.0")
self.assertEqual(find_oldest_supported_version("7.6.0"), "7.6.0")
def test_newer_major_or_minor_versions_not_shown_in_api(self):
# next minor from 7.x
self.assertEqual(find_oldest_supported_version("7.19.0"), "7.19.0")
# next minor from 8.x
self.assertEqual(find_oldest_supported_version("8.12.0"), "8.12.0")
# next patch from 8.8.x
self.assertEqual(find_oldest_supported_version("8.8.3"), "8.8.3")
# next major 9.0
self.assertEqual(find_oldest_supported_version("9.0.0"), "9.0.0")
def test_missing_older_versions_in_api_response(self):
# exists 8.9.1 and 8.8.x, but not 8.9.0
self.assertEqual(find_oldest_supported_version("8.9.0"), "8.9.0")
# exists 8.9.2 and 8.9.4, but not 8.9.3
self.assertEqual(find_oldest_supported_version("8.9.3"), "8.9.3")
def test_or(self):
self.assertEqual(find_oldest_supported_version("8.6.0||8.7.0"), "8.6.0")
self.assertEqual(find_oldest_supported_version("8.9.2||8.9.1||7.17.14"), "7.17.14-SNAPSHOT")
def test_mix(self):
self.assertEqual(find_oldest_supported_version("^8.6.0||~8.7.0"), "8.6.0")
self.assertEqual(find_oldest_supported_version("8.9.2||8.9.1||7.17.14"), "7.17.14-SNAPSHOT")
self.assertEqual(find_oldest_supported_version(
"~8.9.2||>=8.11.0||7.17.14"), "7.17.14-SNAPSHOT")
def test_whitespaces(self):
self.assertEqual(find_oldest_supported_version(" ^8.6.0 || ~8.7.0 "), "8.6.0")
class TestRemoveOperator(unittest.TestCase):
"""Testcase for remove_operator."""
def test_no_operator(self):
self.assertEqual(remove_operator("1.0.0"), "1.0.0")
def test_circumflex(self):
self.assertEqual(remove_operator("^1.0.0"), "1.0.0")
def test_tilda(self):
self.assertEqual(remove_operator("~1.0.0"), "1.0.0")
def test_greater_or_equal(self):
self.assertEqual(remove_operator(">=1.0.0"), "1.0.0")
def test_unknown(self):
with self.assertRaises(Exception):
remove_operator("<=1.0.0")
with self.assertRaises(Exception):
remove_operator("==1.0.0")
class TestHandleOr(unittest.TestCase):
"""Testcase for handle_or."""
def test_single_condition(self):
with self.assertRaises(Exception):
handle_or("0.0.1")
def test_happy_path(self):
# Mock temporarly with the identiy function.
global find_oldest_supported_version
old_func = find_oldest_supported_version
def find_oldest_supported_version(x): return x
self.assertEqual(handle_or("0.1||0.2"), "0.1")
self.assertEqual(handle_or("0.2||0.2.1||0.2.3-alpha"), "0.2")
self.assertEqual(handle_or("1.2||1.2.0||3.2.3-alpha"), "1.2")
# restore mock
find_oldest_supported_version = old_func

@mrodm mrodm mentioned this pull request May 12, 2026
Open
5 tasks
@mrodm mrodm merged commit b15f308 into elastic:main May 12, 2026
8 of 9 checks passed
@mrodm mrodm deleted the add_tests_for_find_oldest_supported_version branch May 12, 2026 09:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@teresaromero teresaromero teresaromero approved these changes

Assignees

@mrodm mrodm

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@mrodm @elasticmachine @teresaromero