YQL-18419 create callback that checks RSS and limits memory usage #4796

lll-phill-lll · 2024-05-23T16:17:04Z

Changelog entry

After this pr it will be possible to set an RSS limit for the process and enable spilling according to current RSS value.
...

Changelog category

New feature

Additional information

...

github-actions · 2024-05-23T16:32:19Z

⚪ 2024-05-23 16:32:18 UTC Pre-commit check for 2835e09 has started.
⚫ 2024-05-23 16:32:36 UTC Check cancelled

github-actions · 2024-05-23T16:51:15Z

⚪ 2024-05-23 16:51:15 UTC Pre-commit check for 5e9e9d3 has started.
⚪ 2024-05-23 16:54:37 UTC Build linux-x86_64-relwithdebinfo is running...
🟢 2024-05-23 17:20:13 UTC Build successful.
⚪ 2024-05-23 17:20:23 UTC Tests are running...
🔴 2024-05-23 17:26:29 UTC Test run completed, no test results found for commit 665a9da. Please check build logs.
⚫ 2024-05-23 17:27:05 UTC Check cancelled

github-actions · 2024-05-23T16:51:34Z

⚪ 2024-05-23 16:51:34 UTC Pre-commit check for 5e9e9d3 has started.
⚪ 2024-05-23 16:54:15 UTC Build linux-x86_64-release-clang14 is running...
🟢 2024-05-23 17:17:22 UTC Build successful.

github-actions · 2024-05-23T16:53:59Z

⚪ 2024-05-23 16:53:59 UTC Pre-commit check for 5e9e9d3 has started.
⚪ 2024-05-23 16:57:20 UTC Build linux-x86_64-release-asan is running...
🟢 2024-05-23 17:24:34 UTC Build successful.
⚪ 2024-05-23 17:24:46 UTC Tests are running...
🔴 2024-05-23 17:26:25 UTC Test run completed, no test results found for commit 665a9da. Please check build logs.
⚫ 2024-05-23 17:26:55 UTC Check cancelled

github-actions · 2024-05-23T17:30:34Z

⚪ 2024-05-23 17:30:33 UTC Pre-commit check for ec4cf5a has started.
⚪ 2024-05-23 17:33:26 UTC Build linux-x86_64-release-asan is running...
🟢 2024-05-23 17:36:29 UTC Build successful.
⚪ 2024-05-23 17:36:46 UTC Tests are running...
🔴 2024-05-23 18:58:47 UTC Test run completed, no test results found for commit a4a82ed. Please check build logs.
⚫ 2024-05-23 18:59:10 UTC Check cancelled

github-actions · 2024-05-23T17:30:42Z

⚪ 2024-05-23 17:30:41 UTC Pre-commit check for ec4cf5a has started.
⚪ 2024-05-23 17:33:36 UTC Build linux-x86_64-release-clang14 is running...
🟢 2024-05-23 17:37:21 UTC Build successful.

github-actions · 2024-05-23T17:33:36Z

⚪ 2024-05-23 17:33:36 UTC Pre-commit check for ec4cf5a has started.
⚪ 2024-05-23 17:36:27 UTC Build linux-x86_64-relwithdebinfo is running...
🟢 2024-05-23 17:39:49 UTC Build successful.
⚪ 2024-05-23 17:40:05 UTC Tests are running...
🔴 2024-05-23 18:58:47 UTC Test run completed, no test results found for commit a4a82ed. Please check build logs.
⚫ 2024-05-23 18:59:07 UTC Check cancelled

github-actions · 2024-05-23T19:01:22Z

⚪ 2024-05-23 19:01:21 UTC Pre-commit check for 899570c has started.
⚪ 2024-05-23 19:04:07 UTC Build linux-x86_64-relwithdebinfo is running...
🟢 2024-05-23 19:21:22 UTC Build successful.
⚪ 2024-05-23 19:21:33 UTC Tests are running...
🔴 2024-05-23 21:16:58 UTC Some tests failed, follow the links below.

Test history

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
73748	61033	0	3	12700	12
⚫ `2024-05-23 21:17:30 UTC` Check cancelled

github-actions · 2024-05-23T19:02:41Z

⚪ 2024-05-23 19:02:41 UTC Pre-commit check for 899570c has started.
⚪ 2024-05-23 19:05:30 UTC Build linux-x86_64-release-clang14 is running...
🟢 2024-05-23 19:21:23 UTC Build successful.

github-actions · 2024-05-23T19:03:48Z

⚪ 2024-05-23 19:03:48 UTC Pre-commit check for 899570c has started.
⚪ 2024-05-23 19:06:40 UTC Build linux-x86_64-release-asan is running...
🟢 2024-05-23 19:24:32 UTC Build successful.
⚪ 2024-05-23 19:24:43 UTC Tests are running...
🔴 2024-05-23 21:16:50 UTC Test run completed, no test results found for commit b038463. Please check build logs.
⚫ 2024-05-23 21:16:57 UTC Check cancelled

This reverts commit 8bdca99.

github-actions · 2024-05-23T21:21:04Z

⚪ 2024-05-23 21:21:03 UTC Pre-commit check for dd5acad has started.
⚪ 2024-05-23 21:24:00 UTC Build linux-x86_64-release-clang14 is running...
🟢 2024-05-23 21:25:14 UTC Build successful.

github-actions · 2024-05-23T21:21:06Z

⚪ 2024-05-23 21:21:05 UTC Pre-commit check for dd5acad has started.
⚪ 2024-05-23 21:24:06 UTC Build linux-x86_64-release-asan is running...
🟢 2024-05-23 21:25:20 UTC Build successful.
⚪ 2024-05-23 21:25:34 UTC Tests are running...
🔴 2024-05-23 23:47:53 UTC Some tests failed, follow the links below.

Test history

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
14743	14598	0	57	70	18

github-actions · 2024-05-23T21:21:08Z

⚪ 2024-05-23 21:21:07 UTC Pre-commit check for dd5acad has started.
⚪ 2024-05-23 21:23:56 UTC Build linux-x86_64-relwithdebinfo is running...
🟢 2024-05-23 21:25:03 UTC Build successful.
⚪ 2024-05-23 21:25:19 UTC Tests are running...
🔴 2024-05-23 23:21:14 UTC Some tests failed, follow the links below.

Test history

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
73748	61037	0	3	12700	8

github-actions · 2024-05-24T08:52:36Z

⚪ 2024-05-24 08:52:36 UTC Pre-commit check for f7f05e9 has started.
⚪ 2024-05-24 08:55:31 UTC Build linux-x86_64-release-clang14 is running...
🟢 2024-05-24 09:11:56 UTC Build successful.

github-actions · 2024-05-24T08:55:13Z

⚪ 2024-05-24 11:42:25 UTC Pre-commit check for f7f05e9 has started.
⚪ 2024-05-24 11:45:12 UTC Build linux-x86_64-release-asan is running...
🟢 2024-05-24 11:46:18 UTC Build successful.
⚪ 2024-05-24 11:46:30 UTC Tests are running...
🔴 2024-05-24 13:49:51 UTC Some tests failed, follow the links below.

Test history

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
14730	14625	0	35	58	12

github-actions · 2024-05-24T08:55:16Z

⚪ 2024-05-24 11:42:36 UTC Pre-commit check for f7f05e9 has started.
⚪ 2024-05-24 11:45:23 UTC Build linux-x86_64-relwithdebinfo is running...
🟢 2024-05-24 11:46:29 UTC Build successful.
⚪ 2024-05-24 11:46:42 UTC Tests are running...
🔴 2024-05-24 13:48:45 UTC Some tests failed, follow the links below.

Test history

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
73747	61038	0	1	12700	8

ydb/library/yql/dq/actors/compute/dq_compute_memory_quota.h

Complements ydb-platform#4796 for sync actor

YQL-18419 create callback that checks RSS and limits memory usage

25817ce

lll-phill-lll requested review from a team as code owners May 23, 2024 16:17

github-actions bot added the new-feature label May 23, 2024

added comment

665a9da

fixup

a4a82ed

fixup

b038463

lll-phill-lll added 2 commits May 24, 2024 00:15

added debug log + spilling

8bdca99

Revert "added debug log + spilling"

f92aa20

This reverts commit 8bdca99.

lll-phill-lll requested a review from Darych May 24, 2024 07:17

fixup

1a531c9

Darych reviewed May 24, 2024

View reviewed changes

ydb/library/yql/dq/actors/compute/dq_compute_memory_quota.h Show resolved Hide resolved

Darych approved these changes May 24, 2024

View reviewed changes

lll-phill-lll merged commit 9c3896d into ydb-platform:main May 24, 2024
8 of 12 checks passed

This was referenced May 27, 2024

Cli changes for autoscaling #4871

Closed

Fix getting mid of range for autoscaling #4884

Closed

Fix getting mid range #4885

Merged

Autoscaling logs #4976

Merged

This was referenced May 29, 2024

YQ-3292 fixed kqprun ResourceBrokerConfig passing #4994

Merged

Added ExecutionId for script execution logs #5014

Merged

niksaveliev mentioned this pull request May 30, 2024

Forbid autoscaling with size retention #5054

Merged

StekPerepolnen mentioned this pull request May 31, 2024

vdisk evict #5093

Merged

This was referenced Jun 6, 2024

Mute split by load ut #5264

Closed

Mute split by load ut #5267

Closed

Mute split by load ut #5272

Merged

Fix build #5310

Merged

serbel324 mentioned this pull request Jun 7, 2024

Don't mark slow disks if there is only one #5340

Merged

niksaveliev mentioned this pull request Jun 10, 2024

Fix split by load ut #5383

Merged

This was referenced Jun 11, 2024

metadata cache keeps merged issues #5461

Merged

json_hc metadata cache #5465

Merged

json tenantinfo status metadata cache #5543

Merged

json query transaction mode #5574

Merged

niksaveliev mentioned this pull request Jun 17, 2024

Fix win build #5622

Closed

yumkam added a commit to yumkam/ydb that referenced this pull request Jun 17, 2024

Install allocator hook with RSS control for sync actor

f2523f1

Complements ydb-platform#4796 for sync actor

This was referenced Jun 19, 2024

hc api doc #5703

Merged

move time difference issue under database level #5859

Merged

metadata requests flags fix #5860

Merged

StekPerepolnen mentioned this pull request Jun 28, 2024

oidc-proxy new iam 2 #6059

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

YQL-18419 create callback that checks RSS and limits memory usage #4796

YQL-18419 create callback that checks RSS and limits memory usage #4796

lll-phill-lll commented May 23, 2024

github-actions bot commented May 23, 2024 •

edited

Loading

github-actions bot commented May 23, 2024 •

edited

Loading

github-actions bot commented May 23, 2024 •

edited

Loading

github-actions bot commented May 23, 2024 •

edited

Loading

github-actions bot commented May 23, 2024 •

edited

Loading

github-actions bot commented May 23, 2024 •

edited

Loading

github-actions bot commented May 23, 2024 •

edited

Loading

github-actions bot commented May 23, 2024 •

edited

Loading

github-actions bot commented May 23, 2024 •

edited

Loading

github-actions bot commented May 23, 2024 •

edited

Loading

github-actions bot commented May 23, 2024 •

edited

Loading

github-actions bot commented May 23, 2024 •

edited

Loading

github-actions bot commented May 23, 2024 •

edited

Loading

github-actions bot commented May 24, 2024 •

edited

Loading

github-actions bot commented May 24, 2024 •

edited

Loading

github-actions bot commented May 24, 2024 •

edited

Loading

YQL-18419 create callback that checks RSS and limits memory usage #4796

YQL-18419 create callback that checks RSS and limits memory usage #4796

Conversation

lll-phill-lll commented May 23, 2024

Changelog entry

Changelog category

Additional information

github-actions bot commented May 23, 2024 • edited Loading

github-actions bot commented May 23, 2024 • edited Loading

github-actions bot commented May 23, 2024 • edited Loading

github-actions bot commented May 23, 2024 • edited Loading

github-actions bot commented May 23, 2024 • edited Loading

github-actions bot commented May 23, 2024 • edited Loading

github-actions bot commented May 23, 2024 • edited Loading

github-actions bot commented May 23, 2024 • edited Loading

github-actions bot commented May 23, 2024 • edited Loading

github-actions bot commented May 23, 2024 • edited Loading

github-actions bot commented May 23, 2024 • edited Loading

github-actions bot commented May 23, 2024 • edited Loading

github-actions bot commented May 23, 2024 • edited Loading

github-actions bot commented May 24, 2024 • edited Loading

github-actions bot commented May 24, 2024 • edited Loading

github-actions bot commented May 24, 2024 • edited Loading

github-actions bot commented May 23, 2024 •

edited

Loading

github-actions bot commented May 23, 2024 •

edited

Loading

github-actions bot commented May 23, 2024 •

edited

Loading

github-actions bot commented May 23, 2024 •

edited

Loading

github-actions bot commented May 23, 2024 •

edited

Loading

github-actions bot commented May 23, 2024 •

edited

Loading

github-actions bot commented May 23, 2024 •

edited

Loading

github-actions bot commented May 23, 2024 •

edited

Loading

github-actions bot commented May 23, 2024 •

edited

Loading

github-actions bot commented May 23, 2024 •

edited

Loading

github-actions bot commented May 23, 2024 •

edited

Loading

github-actions bot commented May 23, 2024 •

edited

Loading

github-actions bot commented May 23, 2024 •

edited

Loading

github-actions bot commented May 24, 2024 •

edited

Loading

github-actions bot commented May 24, 2024 •

edited

Loading

github-actions bot commented May 24, 2024 •

edited

Loading