Support Posix Shared Memory across containers in a pod

[CsatariGergely]

Docker implemented modifiable shmsize (see 1) in version 1.9. It should be possible to define the shmsize of a pod on the API and Kubernetes shall pass this information to Docker.

[Random-Liu]

Also ref #24588 (comment), in which we also discussed whether we should expose shmsize in pod configuration.

[janosi]

I am not sure I can see that discussion in that issue about exposing ShmSize on the Kubernetes API :( As I understand, that discussion is about how to use the Docker API after it introduced the ShmSize attribute.

[Random-Liu]

I would like kube to set an explicit default ShmSize using the option 1 proposed by @Random-Liu and I wonder if we should look to expose ShmSize as a per container option in the future.

I should say "in which we also mentioned whether we should expose shmsize in container configuration."

[janosi]

@Random-Liu All right, thank you! I missed that point.

[dims]

@janosi @CsatariGergely - the 64m default is not enough? what would be the best way to make it configurable for your use? (pass a parameter in kubelet command line?)

[janosi]

@dims Or maybe it is too much to waste? ;)
But yes, sometimes 64m is not enough.
We would prefer a new optional attribute for the pod in PodSpec in the API,like e.g. "shmSize".
As shm is shared among containers in the pod, PodSpec would be the appropriate place, I think.

[janosi]

We have a chance to work on this issue now. I would like to align the design before applying it on the code. Your comments are welcome!

The change on the versioned API would be, that there would be a new field in type PodSpec:

//Optional: Docker "--shm-size" support. Defines the size of /dev/shm in a Docker-managed cotainer 
//If not defined here Docker uses a default value
//Cannot be updated
ShmSize *resource.Quantity `json:"shmSize,omitempty"`

[ddysher]

@janosi Did you have a patch for this? we currently hit this issue running db on k8s, would like to have shm size configurable.

[janosi]

@ddysher We are working on it. We send the PRs in the next weeks.

[wstrange]

Just want to chime in that we are hitting this problem as well

[gjcarneiro]

Hi, is there any known workaround for this problem? I need to increase the shmem size to at least 2GB, and I have no idea how.

[janosi]

@ddysher @wstrange @gjcarneiro Please share your use cases with @vishh and @derekwaynecarr on the pull request #34928 They have concerns about extending the API with this shmsize option, and they have different solution proposals. They would like to understand whether users really require this on the API, or shm size could be adjusted by k8s automatically to some calculated value.

[gjcarneiro]

My use case is a big shared memory database, typically at a 1 GiB order, but we usually reserve 3 GiB shared memory space just in case it grows. This data is constantly being updated by a writer (a process), and must be made available to readers (other processes). Previously we tried redis server for this, but the performance for this solution was not great, so shared memory it is.

My current workaround is (1) mount a tmpfs volume in /dev/shm, as in this openshift article, and (2) make the writer and reader processes all run in the same container.

[wstrange]

My use case is an Apache policy agent plugin that allocates a very large (2GB) cache. I worked around it by setting a very low shm value. This is OK for development, but I need a solution for production.

Adjusting shm size dynamically seems tricky. From my perspective, declaring it as a container resource would be fine.

[ddysher]

My use case is to run database application on top of kubernetes that needs at least (2GB) shared memory. Right now, we just set a large default; it would be nice to have a configurable option.

[vishh]

@ddysher @wstrange @gjcarneiro Do you applications dynamically adjust their behavior based on the shm size? Will they be able to function if the default size is >= pod's memory limit?

[wstrange]

The shm size is configurable only when the application starts (i.e. , you can say "only use this much shm").

It can not be adjusted dynamically.

[vishh]

@wstrange Thanks for clarifying.

[ddysher]

@vishh We have the same case as @wstrange. shm size doesn't need to be adjusted dynamically.

[gjcarneiro]

Same for me, shm size is a constant in a configuration file.

[vishh]

Great. In that case, kubelet can set the default size of /dev/shm to be that of the pod's memory limit. Apps will have to be configured to use a value that is lesser than the pod's memory limit for shm.

[elyscape]

@vishh What about if there is no memory limit imposed on the application? For reference, it looks like Linux defaults to half of the total RAM.

[janosi]

@vishh you can close the PR is you think so.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[arunmk]

/remove-lifecycle stale

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[ijumps]

/remove-lifecycle stale

[bhack]

Any news on this?

[tahayk]

Any news on this? 2

[jerryyifei]

Any update on this :) I'm using helm chart to install containers, and want to figure out where to add shmsize value

[axelborja]

The following may help to implement this feature request easily for those who use containerd as CRI runtime.: containerd/containerd#3655 ;)

[YesterdayxD]

FYI, this is the configuration we're using for k8s:

volumes: [{
    name: 'dshm',
    emptyDir: {
        medium: 'Memory',
        sizeLimit: '256Mi'
    }
}]

maybe the 256Mi can't limit shmsize actually.
The actual size is half of memory size of node where the pod is running on.
Right?

[gaocegege]

FYI, this is the configuration we're using for k8s:

volumes: [{
    name: 'dshm',
    emptyDir: {
        medium: 'Memory',
        sizeLimit: '256Mi'
    }
}]

maybe the 256Mi can't limit shmsize actually.
The actual size is half of memory size of node where the pod is running on.
Right?

I think so and it is why we need #63641

[bashimao]

While this solves the initial problem, at least the current implementation is not acceptable. If any application running in the container tries to go beyond this SHM limit, it is allowed to do so, but within a few seconds the pod is evicted by k8s (see below). Instead, what you want should be a hard limit, such that the operating system in the container throws a no space left on device error, but otherwise continues to run.

  Normal   Started  12m    kubelet, x_machine_x  Started container x_pod_x
  Warning  Evicted  9m38s  kubelet, x_machine_x  Usage of EmptyDir volume "dshm" exceeds the limit "1Gi".
  Normal   Killing  9m38s  kubelet, x_machine_x  Stopping container x_pod_x

[LzyRapx]

volumeMounts:
 - mountPath: /dev/shm
      name: dshm

volumes:
 - name: dshm
      emptyDir:
        medium: Memory

This would be work for me.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[jwm]

/remove-lifecycle stale