New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable ConfigMaps to store binary files as well as character files. #32432

Closed
derekmahar opened this Issue Sep 10, 2016 · 70 comments

Comments

@derekmahar
Contributor

derekmahar commented Sep 10, 2016

Is this a BUG REPORT or FEATURE REQUEST?

FEATURE REQUEST

Please enable ConfigMaps to store binary files as well as character files.

Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"3", GitVersion:"v1.3.6", GitCommit:"ae4550cc9c89a593bcda6678df201db1b208133b", GitTreeState:"clean", BuildDate:"2016-08-26T18:13:23Z", GoVersion:"go1.6.2", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"3", GitVersion:"v1.3.6+coreos.0", GitCommit:"f6f0055b8e503cbe5fb7b6f1a2ee37d0f160c1cd", GitTreeState:"clean", BuildDate:"2016-08-29T17:01:01Z", GoVersion:"go1.6.2", Compiler:"gc", Platform:"linux/amd64"}

Environment:

  • Cloud provider or hardware configuration: HP EliteOne 800 G1 AiO
  • OS (e.g. from /etc/os-release):
NAME="Ubuntu"
VERSION="16.04.1 LTS (Xenial Xerus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 16.04.1 LTS"
VERSION_ID="16.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
UBUNTU_CODENAME=xenial
  • Kernel (e.g. uname -a) (kubectl client): Linux derek-HP-EliteOne-800-G1-AiO 4.4.0-36-generic #55-Ubuntu SMP Thu Aug 11 18:01:55 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
  • Target environment is a three node Vagrant (VirtualBox) cluster running CoreOS 1164.1.0.

What happened?

After creating a ConfigMap from a directory containing a binary Java keystore file, command kubctl describe configmap reported a size for the binary file that was larger than the original file:

derek@derek-HP-EliteOne-800-G1-AiO~:$ kubectl describe configmap platinum-fix-brvm-config
Name:       platinum-fix-brvm-config
Namespace:  default
Labels:     <none>
Annotations:    <none>

Data
====
brvm-fix44.xml: 334363 bytes
keystore.jks:   2266 bytes
settings.txt:   910 bytes
derek@derek-HP-EliteOne-800-G1-AiO~:$ ls -l keystore.jks 
-rw-rw-r-- 1 derek derek 1326 Sep  9 17:52 keystore.jks

When run inside a container having a mounted volume created from the ConfigMap, command ls --format=long --dereference keystore.jks also reported a file size larger than the original:

-rw-r--r--. 1 root root 2266 Sep 10 00:58 keystore.jks

A Java application in the container that attempted to read keystore.jks reported an exception claiming that the file was invalid.

What did you expect to happen?

  • Kubernetes reports the size of the binary file in the ConfigMap as the same as the original file.
  • Kubernetes container log reports the size of the binary file in the ConfigMap as the same as the original file.
  • Java application successfully reads binary file keystore.jks.
  • Result of command file keystore.jks should be keystore.jks: Java KeyStore.

How to reproduce it (as minimally and precisely as possible):

  1. Create a ConfigMap from a directory that contains a Java keystore file (or some other binary file).
  2. Run kubectl describe configmap and observe the file size.
  3. Compare the file size in the ConfigMap to the original file size.
  4. Map the ConfigMap to a volume, mount the volume in a container, and run ls --format=long --dereference keystore.jks and compare the file size of the result to the original file size.
@adohe

This comment has been minimized.

Member

adohe commented Sep 10, 2016

@kubernetes/kubectl

@dims

This comment has been minimized.

Member

dims commented Sep 10, 2016

@derekmahar how exactly did you create the configmap from the files? Can you please share the json/yaml or command line that you used please?

@derekmahar

This comment has been minimized.

Contributor

derekmahar commented Sep 10, 2016

@dims

kubectl create configmap platinum-fix-brvm-config --from-file=$HOME/path/to/directory

Please see http://stackoverflow.com/questions/39420102/does-kubernetes-support-a-binary-file-in-a-configmap for additional details.

@derekmahar

This comment has been minimized.

Contributor

derekmahar commented Sep 10, 2016

Forgot to report that my target environment is a three node Vagrant (VirtualBox) cluster running CoreOS 1164.1.0.

@smarterclayton

This comment has been minimized.

Contributor

smarterclayton commented Sep 10, 2016

In all likelihood the size of the string is different because we're reading arbitrary bytes and shoving them into a string and counting the size of the string as a string instead of a byte array. We may also be reading the config map for write in an inconsistent way. Would have to look, but I suspect that's the root issue here.

@derekmahar

This comment has been minimized.

Contributor

derekmahar commented Sep 11, 2016

On 10 Sep 2016 7:12 pm, "Clayton Coleman" notifications@github.com wrote:

In all likelihood the size of the string is different because we're
reading arbitrary bytes and shoving them into a string and counting the
size of the string as a string instead of a byte array. We may also be
reading the config map for write in an inconsistent way. Would have to
look, but I suspect that's the root issue here.

@smarterclayton On Monday, I'll run a test using an container that hashes a binary file mounted
from a ConfigMap volume and compare this to the hash of the original file.

@liggitt

This comment has been minimized.

Member

liggitt commented Sep 11, 2016

config maps store data as string, not []byte... not sure I'd expect to be able to put arbitrary binary content in them

@derekmahar

This comment has been minimized.

Contributor

derekmahar commented Sep 11, 2016

config maps store data as string, not []byte... not sure I'd expect to be able to put arbitrary binary content in them

@liggitt Do ConfigMaps not encode binary content as strings? If they don't, they
could encode it in base64 format. Binary config file support would allow
the convenient transfer and storage of non-textual configuration files such
as keystores.

How do Kubernetes users typically transfer and store such binary files now?

Should I change this issue from a bug report to a feature request?

@derekmahar

This comment has been minimized.

Contributor

derekmahar commented Sep 12, 2016

Not surprisingly, after running a container that hashes a binary Java keystore file mounted from a ConfigMap volume containing this file, the hash of the keystore file in the volume did not match that of the original keystore file.

derek@derek-HP-EliteOne-800-G1-AiO:~/Projects/keystore-configmap$ kubectl create configmap keystore-configmap --from-file=config/
configmap "keystore-configmap" created
derek@derek-HP-EliteOne-800-G1-AiO:~/Projects/keystore-configmap$ kubectl describe configmap keystore-configmap
Name:       keystore-configmap
Namespace:  default
Labels:     <none>
Annotations:    <none>

Data
====
keystore.jks:   2266 bytes
derek@derek-HP-EliteOne-800-G1-AiO:~/Projects/keystore-configmap$ kubectl create -f keystore-configmap-pod.yaml 
pod "keystore-configmap-pod" created
derek@derek-HP-EliteOne-800-G1-AiO:~/Projects/keystore-configmap$ kubectl logs keystore-configmap-pod keystore
7baa3ccc925a923dcb8ebcbf18673e592928b8c0635ffd1b06ed366bebc79abc  /config/keystore.jks
derek@derek-HP-EliteOne-800-G1-AiO:~/Projects/keystore-configmap$ sha256sum config/keystore.jks 
ee8952c7f39c5a6f5fe8eccfdefe953fd4f6cf206071cf71312447f5eae79663  config/keystore.jks

keystore-configmap-pod.yaml

apiVersion: v1
kind: Pod
metadata:
  name: keystore-configmap-pod
spec:
  containers:
  - name: keystore
    image: ubuntu
    volumeMounts:
    - name: keystore-configmap-volume
      mountPath: /config
    command: [ "sha256sum", "/config/keystore.jks" ]

  volumes:
  - name: keystore-configmap-volume
    configMap:
      name: keystore-configmap
@liggitt

This comment has been minimized.

Member

liggitt commented Sep 12, 2016

@liggitt Do ConfigMaps not encode binary content as strings?

they do not, they store strings. base64-encoding could be layered on top with application logic if desired

@derekmahar

This comment has been minimized.

Contributor

derekmahar commented Sep 12, 2016

they do not, they store strings. base64-encoding could be layered on top with application logic if desired

Yes, or ConfigMaps might store both byte arrays and strings. :) I'll change this from a bug report to a feature request and keep my fingers crossed.

@derekmahar derekmahar changed the title from ConfigMap stores binary file incorrectly. to Enable ConfigMaps to store binary files as well as character files. Sep 12, 2016

@smarterclayton

This comment has been minimized.

Contributor

smarterclayton commented Sep 12, 2016

If JSON is forcing UTF-8 escaping on either end we'll lose binary contents
(not positive that's the case, but it's likely the encodings aren't 100%
full fidelity). In which case ConfigMaps are broken for binary files in
practice if not by design. Protobuf doesn't have this issue, but I don't
think we'll require proto to store / retrieve them.

On Mon, Sep 12, 2016 at 12:06 PM, Derek Mahar notifications@github.com
wrote:

they do not, they store strings. base64-encoding could be layered on top
with application logic if desired

Yes, or ConfigMaps might store both byte arrays and strings. :) I'll
change this from a bug report to a feature request and keep my fingers
crossed.


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#32432 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABG_p_7dzBCLJohI7ANzKl5NQwNUwG2Wks5qpXiRgaJpZM4J5mlO
.

@zreigz

This comment has been minimized.

Contributor

zreigz commented Sep 13, 2016

This feature request looks tempting. If you don't mind I would like start working on this and create PR

@derekmahar

This comment has been minimized.

Contributor

derekmahar commented Sep 13, 2016

One benefit of this feature is that it would obviate the "storage sidecar" container that Java Web Application with Tomcat and Sidercar Container describes. It would represent a kind of prepopulated volume that the sidecar container achieves. I think this is closer to my use case than that of a ConfigMap that supports binary files, though a ConfigMap that accepts binary files would certainly be useful.

Have there been any other discussions about the concept of a prepopulated volume?

@liggitt

This comment has been minimized.

Member

liggitt commented Sep 13, 2016

If you don't mind I would like start working on this and create PR

It might be good to first discuss how you would add support for binary content in a backwards-compatible way, given the current shape of the object. @pmorie might have thoughts

@derekmahar

This comment has been minimized.

Contributor

derekmahar commented Sep 13, 2016

Have there been any other discussions about the concept of a prepopulated volume?

#831 introduces the idea of a "container volume" which seems similar to the concept of a prepopulated volume.

@zreigz

This comment has been minimized.

Contributor

zreigz commented Sep 13, 2016

Yes sure I can prepare some proposition first. Any ideas or tips would be also helpful.

@zreigz

This comment has been minimized.

Contributor

zreigz commented Sep 13, 2016

cc @cheld

@zreigz

This comment has been minimized.

Contributor

zreigz commented Sep 14, 2016

I have prepared very simple solution. I've tested it and works well. This is my proposal:

Creating ConfigMap

  • binary file is recognized and encoded to base64

Consuming ConfigMap via volume plugin

  • checks if data payload is in base64

    yes: checks if it is binary format and return decoded binary data
    no: return original content (string)

I've created my branch for it: zreigz@d9675c7

If you want to test it in local cluster:

$ git clone https://github.com/zreigz/kubernetes.git
$ cd kubernetes
$ git checkout configmap-binary-base64
$ make quick-release
$ cd cluster/images/hyperkube/
$ make build VERSION=v1.10.0

Then clone kube-deploy project to create local k8s cluster with image you already created

$ git clone https://github.com/kubernetes/kube-deploy.git
$ cd kube-deploy/docker-multinode
$ sudo env "PATH=$PATH" K8S_VERSION=v1.10.0 IP_ADDRESS=${HOST IP ADDRESS} ./master.sh
@liggitt

This comment has been minimized.

Member

liggitt commented Sep 14, 2016

how do you tell the difference between binary data and a string that happens to fit the base64 alphabet (like a randomly generated password dGVzAGluAA)?

@derekmahar

This comment has been minimized.

Contributor

derekmahar commented Sep 14, 2016

how do you tell the difference between binary data and a string that happens to fit the base64 alphabet (like a randomly generated password dGVzAGluAA)?

@liggitt Seems @zreigz introduced a package binarycontent that contains a function IsBinaryData(data []byte) and a table binaryFlags that implements the magic algorithm in http://tools.ietf.org/html/draft-abarth-mime-sniff-06#page-8. However, I'll admit that I neither understand the algorithm nor how the table implements the algorithm. Seems very clever if it works.

@zreigz

This comment has been minimized.

Contributor

zreigz commented Sep 14, 2016

In binary data you always have at least one zero (null) byte. I use more efficient way to recognize binary data which is also used in webkit engine: zreigz@d9675c7#diff-eac69e1e373abb6a3689e446c249beb8R40

I do this checks twice. First when I save data to configmap and second when mount file from configmat. When it is not binary data like you mentioned (some random base64 string) then it's stored as string without any modifications. So the most important in my solution is to recognize binary data.

@liggitt

This comment has been minimized.

Member

liggitt commented Sep 14, 2016

You might be able to recognize it on the way in, but binary content encoded as a base64 string is indistinguishable from a literal string that happens to be a base64 encoding of a binary value, so there's no way for the kubelet to know whether it should write the literal string value or base64-decode and write the binary value into the volume. You would need additional information in the object to indicate to the kubelet a particular key should be base64-decoded.

@derekmahar

This comment has been minimized.

Contributor

derekmahar commented Sep 14, 2016

@liggitt Is base64.StdEncoding.DecodeString(data) on line zreigz@d9675c7#diff-ec8effa8c2c7af6d8358e85aa155ab17R287 clever enough to make that distinction? It seems that any base64 encoded content would suffer from this problem unless the decoder knows in advance that the content is encoded.

@smarterclayton

This comment has been minimized.

Contributor

smarterclayton commented Sep 14, 2016

To put it another way, if you add this behavior anyone who is relying on
base64 encoding of a config map not being altered would be broken when you
introduced automatic decoding of their base64 script. I.e. someone who
does:

kubectl create configmap --from-literal=a=$(base64enc some file)

and

kubectl get configmap foo --template='{{ .data.a }}' | base64dec

is now broken.

On Wed, Sep 14, 2016 at 10:35 AM, Jordan Liggitt notifications@github.com
wrote:

You can recognize it on the way in, but binary content encoded as a base64
string is indistinguishable from a literal string that happens to be a
base64 encoding of a binary value.


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#32432 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABG_pzqZ-oeHZRSygUk1daIJddRIWKroks5qqAY2gaJpZM4J5mlO
.

@liggitt

This comment has been minimized.

Member

liggitt commented Sep 14, 2016

@liggitt Is base64.StdEncoding.DecodeString(data) on line zreigz/kubernetes@d9675c7#diff-ec8effa8c2c7af6d8358e85aa155ab17R287 clever enough to make that distinction?

It's an information problem. The same string could be interpreted as a literal string or as a base64 encoded binary value. Without separately recording the intent in the object, there is no process clever enough to decide what the user wanted purely from the string value.

@derekmahar

This comment has been minimized.

Contributor

derekmahar commented Sep 14, 2016

It's an information problem. The same string could be interpreted as a literal string or as a base64 encoded binary value. Without separately recording the intent in the object, there is no process clever enough to decide what the user wanted purely from the string value.

@liggitt Yes, I see your point.

@kargakis

This comment has been minimized.

Member

kargakis commented Jun 24, 2017

/sig node

@DreadPirateShawn

This comment has been minimized.

DreadPirateShawn commented Jul 21, 2017

Curious, any update on this? Mainly, has it been dormant simply because it's not a priority yet, or is there reluctance to use this general approach in the first place?

We're running into use-cases for binary content too, and we're wondering if this is on the radar or if we should assume that the solution won't be from the configmap angle.

@fejta-bot

This comment has been minimized.

fejta-bot commented Jan 1, 2018

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta.
/lifecycle stale

@DreadPirateShawn

This comment has been minimized.

DreadPirateShawn commented Jan 2, 2018

/remove-lifecycle stale

This seems like a still-open gap. (And there's an argument to be made that it's a bug, not just a feature request.)

There seemed to be initial support for fixing this. Would it be possible to get clarification whether it's still on the radar, or if there's no plan to support binary config files?

(Or has it been fixed, simply without a PR linked to this ticket? Or is there a recommended alternative?)

@dims

This comment has been minimized.

Member

dims commented Jan 5, 2018

/lifecycle frozen

@liggitt liggitt added this to the v1.10 milestone Jan 19, 2018

k8s-merge-robot added a commit that referenced this issue Jan 26, 2018

Merge pull request #57938 from dims/add-binary-configmap
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

Add binary configmap

Reviving code from #33549 submitted by @zreigz

**What this PR does / why we need it**:
Add support for binary files in ConfigMap

**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #32432

**Special notes for your reviewer**:

**Release note**:

```release-note
ConfigMap objects now support binary data via a new `binaryData` field. When using `kubectl create configmap --from-file`, files containing non-UTF8 data will be placed in this new field in order to preserve the non-UTF8 data. Use of this feature requires 1.10+ apiserver and kubelets.
```
@serg060606

This comment has been minimized.

serg060606 commented Jul 24, 2018

Hello. Does it mean that will work only from 1.10 kubernetes version ?

@EronWright

This comment has been minimized.

EronWright commented Aug 8, 2018

Is there a follow-up issue to add similar support to the Secret resource?

@liggitt

This comment has been minimized.

Member

liggitt commented Aug 8, 2018

Is there a follow-up issue to add similar support to the Secret resource?

Secret resources have always supported binary content

@bend

This comment has been minimized.

bend commented Sep 7, 2018

I've created a configmap from a zip file but it shows empty in the describe command.

I use this command:

kubectl create configmap openvpn-data -n openvpn --from-file openvpn.zip
kubectl describe configmaps -n openvpn

Name:         openvpn-data                                                                                                                                                              
Namespace:    openvpn                                                                                                                                                                   
Labels:       <none>                                                                                                                                                                    
Annotations:  <none>                                                                                                                                                                    
                                                                                                                                                                                        
Data                                                                                                                                                                                    
====                                                                                                                                                                                    
Events:  <none> 
kubectl get configmaps -n openvpn
NAME           DATA      AGE                                                                                                                                                            
openvpn-data   0         1m  

I use kubernetes version 1.11.1 & 1.11.2 (depending on the node)

@liggitt

This comment has been minimized.

Member

liggitt commented Sep 7, 2018

What does -o yaml show? Might be a bug in the printer

@bend

This comment has been minimized.

bend commented Sep 7, 2018

It shows the file and the data indeed. I can mount it in the same way I mount a configmap volume ?

@liggitt

This comment has been minimized.

Member

liggitt commented Sep 7, 2018

Yes, you can mount it the same way. By default, a config map mount creates a directory at the mount point, containing files for each key.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment