Skip to content

Conversation

@fvanzee
Copy link

@fvanzee fvanzee commented Feb 8, 2025

Fix scheduler arguments to prevent "no space left on device" error

Description

Adding the --etcd-data-dir=/var/lock/dapr/scheduler argument for the scheduler service in the docker-compose example.

Edit 20250211: Increase the tmpfs size from 10000 bytes to 64m. Ealier change setting --etcd-data-dir was only a workaround.

Very simple change to resolve dapr/dapr#8207

Issue reference

dapr/dapr#8207

@fvanzee fvanzee requested review from a team as code owners February 8, 2025 15:12
@fvanzee fvanzee force-pushed the self-hosted-scheduler-docs branch from dcd92b1 to f806e49 Compare February 8, 2025 15:14
msfussell
msfussell previously approved these changes Feb 10, 2025
Copy link
Member

@msfussell msfussell left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@msfussell msfussell dismissed their stale review February 10, 2025 17:26

Need another check

Copy link
Contributor

@cicoyle cicoyle left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Setting --etcd-data-dir helps avoid the immediate no space left on device error by redirecting writes to a different location, but it's more of a workaround. If etcd is consuming significant space, the root issue is likely insufficient storage in the container. Have you tried increasing the volume size or mounting a persistent volume with more space?

@fvanzee
Copy link
Author

fvanzee commented Feb 11, 2025

Setting --etcd-data-dir helps avoid the immediate no space left on device error by redirecting writes to a different location, but it's more of a workaround. If etcd is consuming significant space, the root issue is likely insufficient storage in the container. Have you tried increasing the volume size or mounting a persistent volume with more space?

Yes I did also try to increase the tmpfs size (if you are referring to that) to 10x the size described in the doc, which didn't resolve the issue. I will try to mount a persistent volume, and see if that does work.

@fvanzee
Copy link
Author

fvanzee commented Feb 11, 2025

Alright after some experimenting it does look like the tmpfs size was too low.

It's set to 10000 bytes which is apparently too low for etcd.
Through trial and error I noticed it should be al least set to 64m. So I wasn't even close with my 10x experiment earlier.

I'll update the documentation to set the tmpfs size to 64m to this and remove the --etcd-data-dir argument.

Increase tmpfs size to prevent "no space left on device" error

Signed-off-by: fvanzee <5238563+fvanzee@users.noreply.github.com>
@fvanzee fvanzee force-pushed the self-hosted-scheduler-docs branch from f806e49 to 2cfc002 Compare February 11, 2025 10:45
Copy link
Contributor

@cicoyle cicoyle left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for taking the time to figure this out - Im fine increasing the size to make sure others don't run out of space as well.

Copy link
Member

@msfussell msfussell left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@hhunter-ms hhunter-ms merged commit dea27c7 into dapr:v1.14 Feb 12, 2025
7 checks passed
@marcduiker
Copy link
Contributor

@holopin-bot @fvanzee Thank you! Here's a digital badge as a small token of appreciation.

@holopin-bot
Copy link

holopin-bot bot commented Jun 11, 2025

Congratulations @fvanzee, the maintainer of this repository has issued you a badge! Here it is: https://holopin.io/claim/cmbrzpzfe308407letql1ammz

This badge can only be claimed by you, so make sure that your GitHub account is linked to your Holopin account. You can manage those preferences here: https://holopin.io/account.
Or if you're new to Holopin, you can simply sign up with GitHub, which will do the trick!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Scheduler fails to start when started according to docker compose example in the docs

5 participants