Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: OOM in traceql requests #502

Merged
merged 5 commits into from
May 24, 2024
Merged

fix: OOM in traceql requests #502

merged 5 commits into from
May 24, 2024

Conversation

akvlad
Copy link
Collaborator

@akvlad akvlad commented May 2, 2024

traceql request on more than 10M traces may cause OOM.
The PR should fix the OOM

@metrico metrico deleted a comment from github-actions bot May 2, 2024
@metrico metrico deleted a comment from github-actions bot May 20, 2024
@metrico metrico deleted a comment from github-actions bot May 20, 2024
Copy link

Workflow Telemetry - K6 Test qxip/qryn / Run k6 on qryn:latest

Workflow telemetry for commit 22358f48d0684c899ec664712a813195444152a4
You can access workflow job details here

Step Trace

gantt
	title Run k6 on qryn:latest
	dateFormat x
	axisFormat %H:%M:%S
	Set up job : milestone, 1716218580000, 1716218582000
	Initialize containers : 1716218582000, 1716218601000
	Checkout : 1716218601000, 1716218603000
	Run actions/setup-go@v4 : 1716218603000, 1716218605000
	Download k6 for Prometheus : 1716218605000, 1716218606000
	Download k6 for Prometheus : 1716218606000, 1716218606000
	Collect Workflow Telemetry : 1716218606000, 1716218606000
	Wait for Loki API : 1716218606000, 1716218606000
	Run k6 loki test : 1716218606000, 1716218670000
	Run mv summary.txt loki_summary.txt : 1716218670000, 1716218670000
	Wait for Prom API : 1716218670000, 1716218670000
	Run k6 prometheus test : 1716218670000, 1716218731000
	Run mv summary.txt prom_summary.txt : 1716218731000, 1716218731000
	Wait for Tempo API : 1716218731000, 1716218731000
	Run k6 tempo test : 1716218731000, 1716218791000
	Run mv summary.txt tempo_summary.txt : 1716218791000, 1716218791000
	Strip Text report : 1716218791000, 1716218791000
	Run actions/upload-artifact@v3 : 1716218791000, 1716218791000
	Summary Report : 1716218791000, 1716218791000

Loading

CPU Metrics

chart_stacked_area_time_9b820606-4d50-4298-bfe7-0510df1b4c42

Memory Metrics

chart_stacked_area_time_7434529c-c4b0-4f8d-96c4-98aa7e1678f4

IO Metrics

Read Write
Network I/O chart_line_time_70355311-9761-478b-9f38-38c86c9d050c chart_line_time_9726b462-a89d-4388-8d68-704b65d4b998
Disk I/O chart_line_time_ffaa0c5f-70d9-409c-9d4b-51345c07b2ef chart_line_time_82f7bc3d-2269-43f8-a55b-1e037517db24

Process Trace

Top 100 processes with highest duration

gantt
	title Run k6 on qryn:latest
	dateFormat x
	axisFormat %H:%M:%S

Loading

Copy link

Workflow Telemetry - K6 Test qxip/qryn / Run k6 on qryn:latest

Workflow telemetry for commit aa2babc191107d50aa80799d4b2d2b184f1f4941
You can access workflow job details here

Step Trace

gantt
	title Run k6 on qryn:latest
	dateFormat x
	axisFormat %H:%M:%S
	Set up job : milestone, 1716456030000, 1716456033000
	Initialize containers : 1716456033000, 1716456052000
	Checkout : 1716456052000, 1716456054000
	Run actions/setup-go@v4 : 1716456054000, 1716456054000
	Download k6 for Prometheus : 1716456054000, 1716456055000
	Download k6 for Prometheus : 1716456055000, 1716456055000
	Collect Workflow Telemetry : 1716456055000, 1716456056000
	Wait for Loki API : 1716456056000, 1716456056000
	Run k6 loki test : 1716456056000, 1716456120000
	Run mv summary.txt loki_summary.txt : 1716456120000, 1716456120000
	Wait for Prom API : 1716456120000, 1716456120000
	Run k6 prometheus test : 1716456120000, 1716456182000
	Run mv summary.txt prom_summary.txt : 1716456182000, 1716456182000
	Wait for Tempo API : 1716456182000, 1716456182000
	Run k6 tempo test : 1716456182000, 1716456241000
	Run mv summary.txt tempo_summary.txt : 1716456241000, 1716456241000
	Strip Text report : 1716456241000, 1716456241000
	Run actions/upload-artifact@v3 : 1716456241000, 1716456243000
	Summary Report : 1716456243000, 1716456243000

Loading

CPU Metrics

chart_stacked_area_time_0c8b709e-1fa6-41c1-bfa2-7071d5862ebc

Memory Metrics

chart_stacked_area_time_7097d1ef-c811-4eb9-9854-1df64ec7f254

IO Metrics

Read Write
Network I/O chart_line_time_03a6058d-6266-4dd3-b491-a42f987f7ae5 chart_line_time_16a2f98b-d449-4ffd-ab74-e3c79e1adee8
Disk I/O chart_line_time_f7a3830b-9dff-45b7-a9f3-f44b2d3a7258 chart_line_time_f76e0cd4-cae2-46df-a947-6888b0dbc3c1

Process Trace

Top 100 processes with highest duration

gantt
	title Run k6 on qryn:latest
	dateFormat x
	axisFormat %H:%M:%S

Loading

@akvlad akvlad merged commit f96a7e6 into master May 24, 2024
12 checks passed
@akvlad akvlad deleted the fix/494_axios_error branch May 24, 2024 10:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants