Size limit for ConfigMap #19781
Comments
|
A 1MB cap for an entire ConfigMap is potentially too low. I remember we talked about how to account for the resources (memory/disk) used by secrets, config, etc, in terms of if it is possible to charge it to the namespace vs the Kubelet, but I don't recall if we came to a resolution. |
|
The limit is 1MB because that's the limit for etcd. |
|
Sounds like a 1MB limit for ConfigMap is the ceiling for now then. I'll make a PR. |
|
Do we need a fixed number? We should use something like LimitRange and/or ResourceQuota to cap individual and aggregate resource sizes. Users could attach arbitrary amounts of data to anything given annotations. |
bgrant0607
added
the
priority/backlog
|
Of course, it can't be bigger than what etcd supports, so I'm fine with putting that hard limit in. |
|
I've proposed an extension to Kubernetes for this. Please take a look at #88709. With this extension, you upload your configuration to s3 (or simply git) and reference it in a File resource. Finally the configmap references this file resource. Feel free to upvote that proposal to raise awareness. |
When you create a Kubernetes job, the source files will be uploaded as Kubernetes secrets. However, they have a size limit of 1MB. See: kubernetes/kubernetes#19781 The KubernetesExecutor.run() method has already taken this into account by checking secret file sizes. Yet before invoking run(), at initialization the method register_secret() will be called, which should also check the file size, otherwise the an OSError: (32, 'EPIPE') may be thrown.
When you create a Kubernetes job, the source files will be uploaded as Kubernetes secrets. However, they have a size limit of 1MB. See: kubernetes/kubernetes#19781 The KubernetesExecutor.run() method has already taken this into account by checking secret file sizes. Yet before invoking run(), at initialization the method register_secret() will be called, which should also check the file size, otherwise an OSError: (32, 'EPIPE') may be thrown.
When you create a Kubernetes job, the source files will be uploaded as Kubernetes secrets. However, they have a size limit of 1MB. See: kubernetes/kubernetes#19781 The KubernetesExecutor.run() method has already taken this into account by checking secret file sizes. Yet before invoking run(), at initialization the method register_secret() will be called, which should also check the file size, otherwise an OSError: (32, 'EPIPE') may be thrown.
* 🐛 Fix bug to run Kubernetes jobs with large files When you create a Kubernetes job, the source files will be uploaded as Kubernetes secrets. However, they have a size limit of 1MB. See: kubernetes/kubernetes#19781 The KubernetesExecutor.run() method has already taken this into account by checking secret file sizes. Yet before invoking run(), at initialization the method register_secret() will be called, which should also check the file size, otherwise an OSError: (32, 'EPIPE') may be thrown. * Use old format syntax as Snakemake still supports Python 3.5. * fmt Co-authored-by: Johannes Köster <johannes.koester@uni-due.de>
* 🐛 Fix bug to run Kubernetes jobs with large files When you create a Kubernetes job, the source files will be uploaded as Kubernetes secrets. However, they have a size limit of 1MB. See: kubernetes/kubernetes#19781 The KubernetesExecutor.run() method has already taken this into account by checking secret file sizes. Yet before invoking run(), at initialization the method register_secret() will be called, which should also check the file size, otherwise an OSError: (32, 'EPIPE') may be thrown. * Fix bug: compatibility with git submodules Previously, if the working directory contains git submodles, the function workflow.get_sources will fail. The bug has now been fixed. * Fix bug: handling failed Kubernetes calls Sometimes, when Kubernetes authorization is retried, register_secret() will fail with a 409/conflict ApiException. The exception should be properly handled in a way that the corresponding failed job wont't break the entire main process. * Fix bug: re-authenticate when _wait_for_jobs fails * New feature: node affinity Apply node affinity: label: app=snakemake * New feature: pod's namespace and label * Fix bug: _reauthenticate * Bug fix: process 401 errors when deleting pods * Fix bugs in Kubernetes executor * Tweaks * 🐛 Fix bug to work with git submodules When working with Kubernetes, files returned by "git ls-files" will be deemed as source files and then uploaded to Kubernetes pods as secrets. However, if the working directory is a git repository with submodules, the submodule directory will be returned by "git ls-files" as well, but it's a directory rather than a regular file, so previously snakemake may fail. The bug can be fixed simply by adding "--recurse-submodules" to the git command. * 💥 Set node affinity for Kubernetes pods When start a job via Kubernetes backend, the pod will have node affinity so that it will be scheduled only to a node with the lable: app:snakemake. * 🐛 Fix bug: Kubernetes reauthentication & retry * Fix a minor bug * Reformatting * Minor fix * fix typo * clarity Co-authored-by: Johannes Köster <johannes.koester@uni-due.de>
|
Current etcd version have 1.5mb limit already, with ability to increase it: https://etcd.io/docs/v3.5/dev-guide/limit/
Does this mean, that we can increase this limit in fresh k8s versions too? |
|
this is old ticket. But I hit the size limit issue today when try to pass a large file like 50Mb with configMap. (I understand I can convert the codes with PV, but that needs extra works) Should we not set the limit on configmap, what's the purpose on it? |
|
All kube resources have a limit of about 1 MiB.
…On Mon, Apr 17, 2023, 5:05 PM Bill Wang ***@***.***> wrote:
this is old ticket.
But I hit the size limit issue today when try to pass a large file like
50Mb with configMap.
Should we set the limit on configmap, what's the purpose on it?
—
Reply to this email directly, view it on GitHub
<#19781 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABKWAVHHKPOKLFF5MD7JZADXBXLGJANCNFSM4BY2WB6A>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
Currently we do not enforce any size limits on ConfigMap. The limit for secrets is 1MB. I think 1MB is on the small side for what some people might want, and suggest we consider up to 5MB. The main questions are what the upper limit on a single piece of config data is and what granularity we want to quota total storage on.
@bgrant0607 @smarterclayton @thockin @erictune @eparis @liggitt
cc @kubernetes/rh-cluster-infra
The text was updated successfully, but these errors were encountered: