DAOS-17468 control: Prevent start if transparent hugepages are enabled by tanabarr · Pull Request #16313 · daos-stack/daos

tanabarr · 2025-04-25T11:45:26Z

When THP feature is enabled on linux platforms, SPDK related
hugepage management in DAOS performs sub-optimally. Resulting problems
relate to memory accounting and fragmentation. To remedy, refuse to
start daos_server if THP is enabled on platform and recommend
disabling THP by applying kernel commandline parameters effective on
reboot.

Features: control

Steps for the author:

Commit message follows the guidelines.
Appropriate Features or Test-tag pragmas were used.
Appropriate Functional Test Stages were run.
At least two positive code reviews including at least one code owner from each category referenced in the PR.
Testing is complete. If necessary, forced-landing label added and a reason added in a comment.

After all prior steps are complete:

Gatekeeper requested (daos-gatekeeper added as a reviewer).

github-actions · 2025-04-25T11:49:15Z

Ticket title is 'Prevent start if transparent hugepages are enabled'
Status is 'In Review'
https://daosio.atlassian.net/browse/DAOS-17468

Features: control Signed-off-by: Tom Nabarro <thomas.nabarro@hpe.com>

src/control/fault/code/codes.go

docs/admin/predeployment_check.md

src/control/lib/hardware/sysfs/provider.go

src/control/server/faults.go

src/control/server/server.go

src/control/lib/hardware/sysfs/provider.go

Features: control Signed-off-by: Tom Nabarro <thomas.nabarro@hpe.com>

kjacque

All my issues were addressed. You'll want to fix the typo in the title of the PR (DARS -> DAOS), otherwise looks good.

src/control/server/server.go

tanabarr · 2025-05-16T12:26:08Z

@ryon-jensen @JohnMalmberg can we please ensure that transparent hugepages feature is disabled on all CI test runners. if not it will create problems with DAOS and this PR will cause failures. TIA

…-thp Features: control Allow-unstable-test: true Signed-off-by: Tom Nabarro <thomas.nabarro@hpe.com>

…-thp Features: control Signed-off-by: Tom Nabarro <thomas.nabarro@hpe.com>

daosbuild3 · 2025-06-02T17:01:29Z

Test stage Test RPMs on EL 8.6 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-16313/7/execution/node/1095/log

daosbuild3 · 2025-06-02T18:56:44Z

Test stage Functional on EL 8.8 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-16313/7/execution/node/1086/log

…-thp Features: control Signed-off-by: Tom Nabarro <thomas.nabarro@hpe.com>

daosbuild3 · 2025-06-13T12:58:55Z

Test stage Test RPMs on EL 8.6 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-16313/8/execution/node/1081/log

daosbuild3 · 2025-06-13T14:55:16Z

Test stage Functional on EL 8.8 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-16313/8/execution/node/1095/log

tanabarr · 2025-06-15T21:30:50Z

@ryon-jensen functional tests are failing because presumably on test runner THP is enabled: https://jenkins.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-16313/8/#showFailuresLink I wonder whether THP needs to be enabled on the runner? if we find situations where THP needs to be enabled e.g. VMs then we can add override flag to skip to check.

…-thp Features: control Signed-off-by: Tom Nabarro <thomas.nabarro@hpe.com>

daosbuild3 · 2025-06-24T17:27:10Z

Test stage Test RPMs on EL 8.6 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-16313/9/execution/node/1056/log

daosbuild3 · 2025-06-24T22:19:16Z

Test stage Functional on EL 8.8 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-16313/9/execution/node/1113/log

daosbuild3 · 2025-12-15T13:39:25Z

Test stage Unit Test on EL 8.8 completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-16313/20/testReport/

kjacque

Overall looks good. Nothing I would block on.

docs/admin/deployment.md

daosbuild3 · 2025-12-17T01:31:57Z

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-16313/21/execution/node/1254/log

knard38 · 2025-12-17T08:05:16Z

docs/admin/deployment.md

+```
+
+If `allow_thp: true` parameter is set in server config file global section, the behavior will change
+and the server will start (with THP enabled. SCM tmpfs will be mounted with `huge=always` on `dmg


NIT

Suggested change

and the server will start (with THP enabled. SCM tmpfs will be mounted with `huge=always` on `dmg

and the server will start (with THP enabled). SCM tmpfs will be mounted with `huge=always` on `dmg

will update on follow on or repush

tanabarr · 2025-12-17T08:55:52Z

CI run no. 21 passed all tests apart from one known OSADrain failure. NLT test was skipped in the unit stage. Rerunning only NLT.

daosbuild3 · 2025-12-17T13:03:35Z

Test stage NLT on EL 8.8 completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-16313/23/testReport/

Signed-off-by: Tom Nabarro <thomas.nabarro@hpe.com>

… NLT" This reverts commit 5892623. Signed-off-by: Tom Nabarro <thomas.nabarro@hpe.com>

Signed-off-by: Tom Nabarro <thomas.nabarro@hpe.com>

daosbuild3 · 2025-12-18T08:38:45Z

Test stage Functional Hardware Large MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-16313/26/execution/node/1393/log

This reverts commit 3d55554. Signed-off-by: Tom Nabarro <thomas.nabarro@hpe.com>

Signed-off-by: Tom Nabarro <thomas.nabarro@hpe.com>

Doc-only: false Priority: 2 Signed-off-by: Tom Nabarro <thomas.nabarro@hpe.com>

daosbuild3 · 2025-12-18T17:49:48Z

Test stage NLT on EL 8.8 completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-16313/28/testReport/

tanabarr · 2025-12-19T09:33:03Z

@kjacque @knard38 could I please get reviews again, I've separated out scm_hugepages_disabled concerns from the PR and will handle those separately.
I have requested forced landing for the unrelated NLT container permissions failure.

kjacque

Nothing I'd block on, just some suggestions for a follow-on.

kjacque · 2025-12-19T16:22:20Z

src/control/server/config/server.go

 			seenScmClsIdx = idx
+
+			if seenScmHugeIdx != -1 && scmConf.Scm.DisableHugepages != seenScmHuge {
+				log.Debugf("scm_hugepages_disabled entry %v in %d doesn't match %d",


Might be worth using Error here to log the details, since we error out anyway.

yes, will add in follow-on

kjacque · 2025-12-19T16:25:10Z

src/control/server/config/server_test.go

 			defer test.ShowBufferOnFailure(t, buf)

 			conf := DefaultServer().
+				WithAllowTHP(true). // Enable differences between scm_hugepages_disabled.


Would be useful to have a case where allow_thp: false?

will add in other follow-on although as it's the default value it gets tested in multiple other places

#16313) When THP feature is enabled on linux platforms, SPDK related hugepage management in DAOS performs sub-optimally. Resulting problems relate to memory accounting and fragmentation. To remedy, refuse to start daos_server if THP is enabled on platform and recommend disabling THP by applying kernel commandline parameters effective on reboot. Features: control Signed-off-by: Tom Nabarro <thomas.nabarro@hpe.com>

tanabarr requested review from a team as code owners April 25, 2025 11:45

tanabarr self-assigned this Apr 25, 2025

DAOS-17468 control: Prevent start if transparent hugepages are enabled

5c07867

Features: control Signed-off-by: Tom Nabarro <thomas.nabarro@hpe.com>

tanabarr force-pushed the tanabarr/control-no-thp branch from d46e506 to 5c07867 Compare April 25, 2025 15:16

tanabarr requested a review from a team as a code owner April 25, 2025 15:16

change implementation to query sysfs file

a8b8809

Features: control Signed-off-by: Tom Nabarro <thomas.nabarro@hpe.com>

kjacque requested changes Apr 28, 2025

View reviewed changes

kjacque reviewed Apr 28, 2025

View reviewed changes

src/control/lib/hardware/sysfs/provider.go Outdated Show resolved Hide resolved

src/control/lib/hardware/sysfs/provider.go Outdated Show resolved Hide resolved

address review comments

95f3af4

Features: control Signed-off-by: Tom Nabarro <thomas.nabarro@hpe.com>

tanabarr requested review from kjacque, knard38 and mjmac April 29, 2025 22:12

remove inadvertent change

b4f9a25

Features: control Signed-off-by: Tom Nabarro <thomas.nabarro@hpe.com>

kjacque previously approved these changes Apr 30, 2025

View reviewed changes

src/control/server/server.go Outdated Show resolved Hide resolved

knard38 previously approved these changes May 5, 2025

View reviewed changes

tanabarr added 2 commits May 17, 2025 10:39

Merge remote-tracking branch 'origin/master' into tanabarr/control-no…

25ad978

…-thp Features: control Allow-unstable-test: true Signed-off-by: Tom Nabarro <thomas.nabarro@hpe.com>

Merge remote-tracking branch 'origin/master' into tanabarr/control-no…

7189bab

…-thp Features: control Signed-off-by: Tom Nabarro <thomas.nabarro@hpe.com>

tanabarr changed the title ~~DARS-17468 control: Prevent start if transparent hugepages are enabled~~ DAOS-17468 control: Prevent start if transparent hugepages are enabled Jun 4, 2025

Merge remote-tracking branch 'origin/master' into tanabarr/control-no…

118e8b9

…-thp Features: control Signed-off-by: Tom Nabarro <thomas.nabarro@hpe.com>

Merge remote-tracking branch 'origin/master' into tanabarr/control-no…

dceef3c

…-thp Features: control Signed-off-by: Tom Nabarro <thomas.nabarro@hpe.com>

kjacque previously approved these changes Dec 16, 2025

View reviewed changes

docs/admin/deployment.md Show resolved Hide resolved

knard38 previously approved these changes Dec 17, 2025

View reviewed changes

tanabarr added 2 commits December 17, 2025 15:46

disallow differences in scm_hugepages_disabled across engines

a472149

Signed-off-by: Tom Nabarro <thomas.nabarro@hpe.com>

verify if auto-setting scm_hugepages_disabled to true impacts NLT

5892623

Signed-off-by: Tom Nabarro <thomas.nabarro@hpe.com>

tanabarr dismissed stale reviews from knard38 and kjacque via 5892623 December 17, 2025 16:25

tanabarr added 2 commits December 17, 2025 18:30

Revert "verify if auto-setting scm_hugepages_disabled to true impacts…

98f1897

… NLT" This reverts commit 5892623. Signed-off-by: Tom Nabarro <thomas.nabarro@hpe.com>

enable allow_thp to use scm hugepages in nlt

3d55554

Signed-off-by: Tom Nabarro <thomas.nabarro@hpe.com>

tanabarr added 3 commits December 18, 2025 15:39

Revert "enable allow_thp to use scm hugepages in nlt"

8aecd49

This reverts commit 3d55554. Signed-off-by: Tom Nabarro <thomas.nabarro@hpe.com>

don't touch scm_hugepages_disabled

b63682c

Signed-off-by: Tom Nabarro <thomas.nabarro@hpe.com>

update documentation

73056bf

Doc-only: false Priority: 2 Signed-off-by: Tom Nabarro <thomas.nabarro@hpe.com>

tanabarr requested review from NiuYawei, kjacque and knard38 December 19, 2025 09:29

tanabarr added the forced-landing The PR has known failures or has intentionally reduced testing, but should still be landed. label Dec 19, 2025

knard38 approved these changes Dec 19, 2025

View reviewed changes

kjacque approved these changes Dec 19, 2025

View reviewed changes

daltonbohning requested a review from a team December 19, 2025 17:18

daltonbohning merged commit 7faa4d4 into master Dec 19, 2025
40 of 42 checks passed

daltonbohning deleted the tanabarr/control-no-thp branch December 19, 2025 17:19

tanabarr mentioned this pull request Jan 26, 2026

DAOS-17468 control: Prevent start if transparent hugepages are enable… #17457

Merged

6 tasks

	and the server will start (with THP enabled. SCM tmpfs will be mounted with `huge=always` on `dmg
	and the server will start (with THP enabled). SCM tmpfs will be mounted with `huge=always` on `dmg

Conversation

tanabarr commented Apr 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Steps for the author:

After all prior steps are complete:

Uh oh!

github-actions bot commented Apr 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kjacque left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

tanabarr commented May 16, 2025

Uh oh!

daosbuild3 commented Jun 2, 2025

Uh oh!

daosbuild3 commented Jun 2, 2025

Uh oh!

daosbuild3 commented Jun 13, 2025

Uh oh!

daosbuild3 commented Jun 13, 2025

Uh oh!

tanabarr commented Jun 15, 2025

Uh oh!

daosbuild3 commented Jun 24, 2025

Uh oh!

daosbuild3 commented Jun 24, 2025

Uh oh!

daosbuild3 commented Dec 15, 2025

Uh oh!

kjacque left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

daosbuild3 commented Dec 17, 2025

Uh oh!

knard38 Dec 17, 2025

Choose a reason for hiding this comment

Uh oh!

tanabarr Dec 17, 2025

Choose a reason for hiding this comment

Uh oh!

tanabarr Dec 18, 2025

Choose a reason for hiding this comment

Uh oh!

tanabarr commented Dec 17, 2025

Uh oh!

daosbuild3 commented Dec 17, 2025

Uh oh!

daosbuild3 commented Dec 18, 2025

Uh oh!

daosbuild3 commented Dec 18, 2025

Uh oh!

tanabarr commented Dec 19, 2025

Uh oh!

kjacque left a comment

Choose a reason for hiding this comment

Uh oh!

kjacque Dec 19, 2025

Choose a reason for hiding this comment

Uh oh!

tanabarr Dec 19, 2025

Choose a reason for hiding this comment

Uh oh!

kjacque Dec 19, 2025

Choose a reason for hiding this comment

Uh oh!

tanabarr Dec 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

tanabarr commented Apr 25, 2025 •

edited

Loading

github-actions bot commented Apr 25, 2025 •

edited

Loading

tanabarr Dec 19, 2025 •

edited

Loading