Research a way around non empty new volumes and containers expecting empty volumes

[chengwang86]

As a VIC user, I should be able to add a volume and still run popular container images, such as mysql and postgres.

---

VIC version:

0.8

Deployment details:

What was the vic-machine create command used to deploy the VCH?
bin/vic-machine-linux create --target=10.160.130.42 --user=usr --volume-store=datastore1/test:default --image-store=datastore1 --appliance-iso=bin/appliance.iso --bootstrap-iso=bin/bootstrap.iso --password=pwd --force=true --bridge-network=bridge --compute-resource=/ha-datacenter/host/10.160.106.191/Resources --public-network=vm-network --name=vch-3 --no-tlsverify --no-tls --debug 1

Steps to reproduce:
Run "docker-compose up" with the following yml:

version: '2'
services:
  db:
    image: mysql:5.7
    environment:
      MYSQL_ROOT_PASSWORD: root
    volumes:
      - /var/lib/mysql

Actual behavior:
In the mysql log, I find

--initialize specified but the data directory has files in it.

Expected behavior:
The mysql db should be started successfully.

Additional comment:
Even if I remove "volumes: - /var/lib/mysql", I still observe the same behavior.

[sflxn]

We had a similar problem with postgres containers in compose. Here are the comments I put in the compose file we have in our repo's demo folder:

# Postgres container will create an anonymous volume.  VIC will use a
# vmdk for the volume.  Since it is a new volume, it will have a lost+found
# folder, and the Postgres init scripts do not like a non-empty folder (the
# lost+found folder).  We work around this by directing the init scripts to
# use another arbitrary folder.  We do this by setting the PGDATA env var.
  db:
    container_name: db
    image: postgres:9.4
    environment:
      - PGDATA=/var/lib/postgresql/data/data

[stuclem]

Proposed text for the release notes. @chengwang86 or @sflxn, could you please provide the correct MYSQL env variable to replace the one that I made up in the example? Thanks!

• The mysql image does not work with docker compose up #3857
  If you run docker compose up and the application that you are creating uses the mysql image, the database does not initialize. The MYSQL log contains the error --initialize specified but the data directory has files in it. This happens because the mysql container creates an anonymous volume. vSphere Integrated Containers Engine create a new VDMK for this volume, which contains a lost+found directory.
  Workaround: Specify an arbitrary, empty data folder in a MYSQL environment variable in the YML file.

  db:
    image: mysql:5.7
    environment:
      MYSQL_ROOT_PASSWORD: root
    volumes:
      - /var/lib/mysql/data/data
  

[stuclem]

@chengwang86 told me in a Slack chat that he couldn't get a workaround to work for MySQL. @sflxn, do you know of one?

[stuclem]

Added the following in https://github.com/vmware/vic/releases/tag/v0.8.0:

• The mysql image does not work with docker compose up #3857
  If you run docker compose up and the application that you are creating uses the mysql image, the database does not initialize. The MYSQL log contains the error --initialize specified but the data directory has files in it. This happens because the mysql container creates an anonymous volume. vSphere Integrated Containers Engine creates a new VDMK for this volume, which contains a lost+found directory, whereas mysql requires the volume to be empty.
  Workaround: Currently none.

@chengwang86 and @sflxn is this OK?

[chengwang86]

@stuclem I think this is OK. We can also suggest the customer to use MariaDB, which works fine for docker compose on vic. But this may not seem like a workaround though.

[stuclem]

Thanks @chengwang86. I updated the workaround as follows:

Workaround: Currently none, but it is possible to use the MariaDB container with docker compose up and vSphere Integrated Containers Engine.

Removing the kind/notes flag as this is now in the release notes.

[sflxn]

I don't know of a work around for mysql. I'd have to look at the mysql init script in the container to determine if we can workaround it. I'm not sure simply suggesting people use mariadb is the solution. For new application, that may be fine. Those who have production apps that uses mysql and want to convert over to VIC may not have the choice to simply switch over to mariadb.

It's my opinion that this is a big enough issue for us to try and find a solution instead of asking users to do a workaround. We should investigate one.

I converted this issue to a research spike. We now have two very popular DB, mysql and postgres, that have a problem with the way we mount volumes.

[stuclem]

Thanks @sflxn. Reworded the workaround as follows:

Workaround: Currently none. VMware is working to find a solution. In the meantime, it is possible to use the MariaDB container with docker compose up and vSphere Integrated Containers Engine.

[chengwang86]

@sflxn @stuclem @anchal-agrawal

First, we find that the problem does not exist if the customer uses mysql:5.6 instead of 5.7. See an example here https://blogs.vmware.com/vsphere/2017/02/deploying-wordpress-vsphere-integrated-containers.html.

Second, Anchal found this https://dev.mysql.com/doc/refman/5.7/en/server-options.html#option_mysqld_ignore-db-dir

if a MySQL configuration locates the data directory at the root of a file system on Unix, the system might create a lost+found directory there that the server should ignore. Starting the server with --ignore-db-dir=lost+found causes that name not to be listed as a database.

I have tested mysql:5.7 and mysql:lastest on vic for docker-compose using the following compose file and it worked

version: '2'
services:
  db:
    image: mysql:5.7
    environment:
      MYSQL_ROOT_PASSWORD: root
    command: ["mysqld", "----ignore-db-dir=lost+found"]
    volumes:
      - /var/lib/mysql

If you already have a default volume when creating the VCH, this will mount /var/lib/mysql to your default volume which has a folder for lost+found. With --ignore-db-dir=lost+found, mysql:5.7 ignores this folder.

Alternatively, we can also create an external volume on vic outside of docker-compose, e.g.,

docker -H ... volume create --name aaa

Then mount /var/lib/mysql to "aaa", and it still works if using "--ignore-db-dir"

version: '2'
services:
  db:
    image: mysql:5.7
    environment:
      MYSQL_ROOT_PASSWORD: root
    command: ["mysqld", "----ignore-db-dir=lost+found"]
    volumes:
      - aaa:/var/lib/mysql
volumes:
   aaa:
       external: true

[anchal-agrawal]

See #3909 (comment).

[stuclem]

@chengwang86 , so, as a workaround, I should document that you can use v5.6 or use the ----ignore-db-dir=lost+found option with 5.7. Correct?

[anchal-agrawal]

@stuclem Quick note, the option is --ignore-db-dir=lost+found, not ----ignore-db-dir=lost+found. And yes, those are the workarounds.

[stuclem]

Updated the workarounds as follows:

Workarounds: Use the mysql:5.6 image, which is not subject to this issue. Alternatively, if you are using mysql:5.7, specify --ignore-db-dir=lost+found in the YML file:

version: '2'
   services:
      db:
           image: mysql:5.7
           environment:
              MYSQL_ROOT_PASSWORD: root
           command: ["mysqld", "----ignore-db-dir=lost+found"]
           volumes: 
                 - /var/lib/mysql
  

[chengwang86]

This looks good. You just need to change ---- to -- in the yml file.

…

On Tuesday, February 21, 2017, Stuart Clements ***@***.***> wrote: Updated the workarounds as follows: *Workarounds*: Use the mysql:5.6 image, which is not subject to this issue. Alternatively, if you are using mysql:5.7, specify --ignore-db-dir=lost+found in the YML file: version: '2' services: db: image: mysql:5.7 environment: MYSQL_ROOT_PASSWORD: root command: ["mysqld", "----ignore-db-dir=lost+found"] volumes: - /var/lib/mysql — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#3857 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AVRU4zeyYjg18y2MFPJjQMiBgHR3EgLmks5rewlLgaJpZM4L4vzr> .

[stuclem]

Done in https://github.com/vmware/vic/releases/tag/v0.8.0

[chengwang86]

@sflxn

After discussing with @anchal-agrawal and @matthewavery , it seems like we would have empty volumes if the customer is willing to deploy NFS. See this PR #3926. This could also be a more general workaround.

[matthewavery]

As a note, PR #3926 is only one part of the over arching epic #2303 . NFS support natively will not be available as a feature until #2303 is done in its entirety.

[anchal-agrawal]

Trying to gauge what the output of this issue should be.

@sflxn, we now have fairly good solutions for mysql and postgres. Also, as @matthewavery mentioned, we'll likely be able to support empty volumes with NFS. In that case, the customer would need to deploy the VCH with an NFS endpoint as a volume store.

Are there other solutions? I'm not sure if there's a way to obtain empty volumes with the current volume support. @fdawg4l?

[fdawg4l]

@anchal-agrawal, for my own clarity, I'm going to try to summarize the issue.

You want to attach a volume at /some/directory, but copy the contents of /some/directory into the volume first. Is that right? I think that's what you mean by non-empty. In that case, lets change the subject of the issue because it's very unclear. Something like "maintain image directory contents for new volumes". Just a suggestion.

The answer is (as was mentioned in the original issue) this is non-trivial. There is a lifecycle problem for volumes used in this way. Here are a few examples.

Example 1 - new Image

I have an image with a directory /var/lib/db which includes some schema definitions and such. I want to attach a volume at that directory so I can reuse it (the volume) later with a new container. So, we copy the contents into the new volume, then mount the volume where we expect it. However, now I want to destroy the container and migrate the volume to a different container running the same image. How, in the second container, will we know not to copy the files again into the volume (thereby, possibly, corrupting the volume contents and suffering dataloss)?

Example 2 - upgrade image

Say we do the same as above; we have a db directory and we copy it into a volume. Now we do the same as above and destroy the container. If we create a new container with the same image name but different version whose contents of /var/lib/db are now new, and we attached a volume to it, whats the expectation? What updates the contents of the volume? Do we skip in this case and not copy, do we throw an error? etc. This needs to be answered.

Example 3 - restart

Same again, same image, container, contents in /var/lib/db. Lets say you restart the container. What prevents the restart from copying the original contents of the image (in /var/lib/db) into the new in-use volume?

The naive solution is the first time you attach a volume, you copy the contents of the target directory into it, update some metadata (ON THE VOLUME ITSELF, NOT IN VIC) so you know you copied the data (maybe a .copied file in the root of the volume) and check for this file every time the container attaches the volume so it skips this step. This is the naive solution and will likely work for most use cases. If a user wants to do an upgrade or the like. it's up to them to mutate the data by hand and create that done file so when they reattach the volume, we don't nuke the contents.

[anchal-agrawal]

Thanks @fdawg4l! I didn't mean to conflate this issue with #3482, which fits with your comment.

This issue is regarding mount points having the lost+found folder and applications such as mysql and postgres expecting new volume directories to be empty.

[fdawg4l]

OH! My mistake @anchal-agrawal! I think there's a simple fix for that. Let me get back to you.

[fdawg4l]

The most obvious solution is deleting the file after we create the volume on the vch. But that seems like an unnecessary step. It would involve mkfs.ext4 (like we are now), then mounting the result, rm -rf lost+found, unmounting, then detatching (like we are now). The lost+found directory is created by (from what I can find) ext itself. There is possibly a knob i can pass to mkfs which skips this step since there are no inodes for lost+found to populate any lost items with. fsck will recreate this directory, but we don't fsk our disks (...and maybe we should come to think of it).

I'm still looking for knobs to deal with it transparently, but what I described above will definitely work.

[sflxn]

@chengwang86 @anchal-agrawal good job finding the workaround. Glad to see mysql anticipated this. Put this issue in verify and I'll put it in closed.