Docker with devicemapper driver and dm.thinpooldev lead to data loss #15629

Open
cubbasa opened this Issue Aug 17, 2015 · 20 comments

Comments

Projects
None yet
10 participants
@cubbasa

cubbasa commented Aug 17, 2015

Description
Problem occur when docker is used with devicemapper driver and lvm thin pool provided by dm.thinpooldev.
It's shown on Ubuntu 14.04 and Debian 8 with all docker releases I've checked so far: 1.6.2, 1.7.1, 1.8.1 and 1.8.2.
This problem is related to how docker manage thin pool using devicemapper directly and ignore lvm.

Enviroment:

docker version:
Client version: 1.7.1
Client API version: 1.19
Go version (client): go1.4.2
Git commit (client): 786b29d
OS/Arch (client): linux/amd64
Server version: 1.7.1
Server API version: 1.19
Go version (server): go1.4.2
Git commit (server): 786b29d
OS/Arch (server): linux/amd64
docker info:
Containers: 1
Images: 11
Storage Driver: devicemapper
 Pool Name: store-tpool-tpool
 Pool Blocksize: 65.54 kB
 Backing Filesystem: extfs
 Data file: 
 Metadata file: 
 Data Space Used: 739.7 MB
 Data Space Total: 10.74 GB
 Data Space Available: 9.998 GB
 Metadata Space Used: 761.9 kB
 Metadata Space Total: 12.58 MB
 Metadata Space Available: 11.82 MB
 Udev Sync Supported: false
 Deferred Removal Enabled: false
 Library Version: 1.02.82-git (2013-10-04)
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 3.16.0-30-generic
Operating System: Ubuntu 14.04.2 LTS
CPUs: 1
Total Memory: 993.8 MiB
Name: ubuntu-5
ID: ZFES:GUHI:PMOK:IY4R:ANJW:M4GU:BNW4:IKD2:NHYV:Q2DO:WYQR:T4RL
WARNING: No swap limit support
uname -a
Linux ubuntu-5 3.16.0-30-generic #40~14.04.1-Ubuntu SMP Thu Jan 15 17:43:14 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

To reproduce problem:

  1. Create lvm thin pool using lvcreate or lvconvert
  2. Pass lvm thin pool for exclusive use by docker
  3. Run docker daemon with devicemapper driver and dm.thinpooldev
  4. Import volume to the docker or create new container
  5. Try to extend or make any operation on lvm thin pool using lvm tools like lvextend thin data (Ubuntu 14.04 and Debian 8) or thin meta data (Debian 8) or lvchange/vgchange

You will get error similiar to this:
Thin pool transaction_id=12, while expected: 0

On Ubuntu 14.04 is non fatal You can use still Your lvm thin pool but each operation using lvm tools produce error like mention before.
On Debian 8 this error is fatal and lvm thin pool and all volumes become unavailable. Volumes can't be recovered using lvm thin provisioning tools.

Difference is in lvm tools version which on Ubuntu 14.04 is 2.02.98 and on Debian 8 is 2.02.111. The newer version of lvm tools checking transaction_id and abort futher operation is there mismatch (this safety option was introduced with commit 8f518cf1979e4cbfce40f6ae1bed02bd5b9a5b35 in lvm tools)

The problem exists because of how transaction_is managed by docker and lvm2.
Transaction_id is counter which help devicemapper to maintain integrity of thin pool beetween kernel space and user space managers. Docker use devicemapper directly and increment transaction_id when docker make any changing operation on thin pool like import volume or create container or destroy container which is correct.
Lvm maintain it's own metadata with it's own transaction_id which is checked with version get directly from devicemapper before any operation made using lvm tools like vgchange or lvextend.
This lead to the situation that lvm is not aware of transaction made by docker which use devicemapper directly and this mismach can't be handled by lvm and newer version of lvm abort any operation on thin pool. Docker use devicemapper directly and don't update lvm metadata about operation that where made directly by docker.

Workound
On Debian 8 there is one time workaround which allow you to bring Your thin pool online.
You have to use vgcfgbackup to create recent backup of lvm metadata manually edit metadata file and change transaction_id of thin pool with the current get from devicemapper devicemapper status thinpool and then restore metadata with vgcfgrestore.
The problem is that one time workaround because any operation on docker increment transaction_id and generate mismatch between lvm metadata transaction_id and one with is store by devicemapper.
On Ubuntu 14.04 this workaround is not working because lvm tools are too old and don't support restore of thin pools.

@thaJeztah

This comment has been minimized.

Show comment
Hide comment
@thaJeztah

thaJeztah Aug 17, 2015

Member

Thanks for reporting

ping @rhvgoyal could you have a look at this? thanks!

Member

thaJeztah commented Aug 17, 2015

Thanks for reporting

ping @rhvgoyal could you have a look at this? thanks!

@rhvgoyal

This comment has been minimized.

Show comment
Hide comment
@rhvgoyal

rhvgoyal Aug 18, 2015

Contributor

@cubbasa

I am reading through this now. First thing I noticed is that "Udev sync" is not supported. Please use dynamically built binary so that libdm works well with udev. Otherwise all bets are off.

Can you please use dynamically linked docker and see if you can still reproduce the problem.

Contributor

rhvgoyal commented Aug 18, 2015

@cubbasa

I am reading through this now. First thing I noticed is that "Udev sync" is not supported. Please use dynamically built binary so that libdm works well with udev. Otherwise all bets are off.

Can you please use dynamically linked docker and see if you can still reproduce the problem.

@rhvgoyal

This comment has been minimized.

Show comment
Hide comment
@rhvgoyal

rhvgoyal Aug 18, 2015

Contributor

Only one entity can create thin devices in pool. Either lvm or docker. If docker has taken the ownership of thin pool, then lvm commands should exit gracefully and not cause any loss of data or corruption. I thought all that has been fixed. May be it is just a case of old lvm tools version.

Contributor

rhvgoyal commented Aug 18, 2015

Only one entity can create thin devices in pool. Either lvm or docker. If docker has taken the ownership of thin pool, then lvm commands should exit gracefully and not cause any loss of data or corruption. I thought all that has been fixed. May be it is just a case of old lvm tools version.

@cubbasa

This comment has been minimized.

Show comment
Hide comment
@cubbasa

cubbasa Aug 18, 2015

The data loss I've mention is that Your docker volume become unavailable and can't be accessed and any normal way and tool can't help You like thin_repair. But with my workaround You can bring online volume again but is really ugly hack and it's took me some time to get solution to get my volume back. Before I get this hack I've lost about 20-30 volumes to get this solution.

cubbasa commented Aug 18, 2015

The data loss I've mention is that Your docker volume become unavailable and can't be accessed and any normal way and tool can't help You like thin_repair. But with my workaround You can bring online volume again but is really ugly hack and it's took me some time to get solution to get my volume back. Before I get this hack I've lost about 20-30 volumes to get this solution.

@rhvgoyal

This comment has been minimized.

Show comment
Hide comment
@rhvgoyal

rhvgoyal Aug 18, 2015

Contributor

Ok, first thing first. When I was doing testing, I realized that we need atleast lvm tools version lvm2-2.02.112 for this to work properly. I have put a comment in docker-storage-setup.

https://github.com/projectatomic/docker-storage-setup/blob/master/docker-storage-setup.sh#L54

So please make sure you are running atleast version 112 and retry.

I tried deactivating thin pool on my system when a container was running and it failed.

[root@vm2-f22 fedora]# lvchange -an fedora/docker-pool
Logical volume fedora/docker-pool is used by another device.

But if none of the containers was running, it succeeded. But I activated it back and docker was working still fine and no data was lost.

I am wondering can docker keep open the thin pool device so that lvm tools will find it busy and not be able to do their regular operations.

Contributor

rhvgoyal commented Aug 18, 2015

Ok, first thing first. When I was doing testing, I realized that we need atleast lvm tools version lvm2-2.02.112 for this to work properly. I have put a comment in docker-storage-setup.

https://github.com/projectatomic/docker-storage-setup/blob/master/docker-storage-setup.sh#L54

So please make sure you are running atleast version 112 and retry.

I tried deactivating thin pool on my system when a container was running and it failed.

[root@vm2-f22 fedora]# lvchange -an fedora/docker-pool
Logical volume fedora/docker-pool is used by another device.

But if none of the containers was running, it succeeded. But I activated it back and docker was working still fine and no data was lost.

I am wondering can docker keep open the thin pool device so that lvm tools will find it busy and not be able to do their regular operations.

@cubbasa

This comment has been minimized.

Show comment
Hide comment
@cubbasa

cubbasa Aug 18, 2015

From lvm WHATS_NEW

Version 2.02.112 - 11th November 2014
.....
Inital support for external users of thin pools based on transaction_id
....

For curiosity this option is needed to get proper work?

cubbasa commented Aug 18, 2015

From lvm WHATS_NEW

Version 2.02.112 - 11th November 2014
.....
Inital support for external users of thin pools based on transaction_id
....

For curiosity this option is needed to get proper work?

@rhvgoyal

This comment has been minimized.

Show comment
Hide comment
@rhvgoyal

rhvgoyal Aug 18, 2015

Contributor

Yes.

Contributor

rhvgoyal commented Aug 18, 2015

Yes.

@rhvgoyal

This comment has been minimized.

Show comment
Hide comment
@rhvgoyal

rhvgoyal Aug 18, 2015

Contributor

I tried creating a new thin device and it fails.

[root@vm2-f22 fedora]# lvcreate -n vivek-thin -V 1G --thinpool fedora/docker-pool
Cannot use thin pool fedora/docker-pool with transaction id 493 for thin volumes. Expected transaction id 0.

That's the intent. Once docker has taken over the ownership of the pool, it will manage transaction id and lvm will bail out.

Contributor

rhvgoyal commented Aug 18, 2015

I tried creating a new thin device and it fails.

[root@vm2-f22 fedora]# lvcreate -n vivek-thin -V 1G --thinpool fedora/docker-pool
Cannot use thin pool fedora/docker-pool with transaction id 493 for thin volumes. Expected transaction id 0.

That's the intent. Once docker has taken over the ownership of the pool, it will manage transaction id and lvm will bail out.

@rhvgoyal

This comment has been minimized.

Show comment
Hide comment
@rhvgoyal

rhvgoyal Aug 18, 2015

Contributor

So once you upgrade to -112, can you give me a list of lvm commands which make you loose data.

Contributor

rhvgoyal commented Aug 18, 2015

So once you upgrade to -112, can you give me a list of lvm commands which make you loose data.

@cubbasa

This comment has been minimized.

Show comment
Hide comment
@cubbasa

cubbasa Aug 18, 2015

  1. I've create thin pool using lvcreate
  2. Run docker with dm.thinpooldev
  3. I've extended lvm thin pool using lvextend (it succeded)
  4. On Debian lvm can't bring online again volumes after they were suspendend before/during lvextend. After I've tried manually run vgchange -ay they become totally unavailable.
    On Ubuntu I still could use volume but each operation using lvm tools even vgdisplay, lvdisplay generate error I've mention before.
    On Ubuntu saver is that lvm tools are so old that are before 8f518cf1979e4cbfce40f6ae1bed02bd5b9a5b35 commit that way there is only error but is not fatal. Lvm ignore mismach between transaction id from lvm and devicemapper but Debian users don't have so much luck.

cubbasa commented Aug 18, 2015

  1. I've create thin pool using lvcreate
  2. Run docker with dm.thinpooldev
  3. I've extended lvm thin pool using lvextend (it succeded)
  4. On Debian lvm can't bring online again volumes after they were suspendend before/during lvextend. After I've tried manually run vgchange -ay they become totally unavailable.
    On Ubuntu I still could use volume but each operation using lvm tools even vgdisplay, lvdisplay generate error I've mention before.
    On Ubuntu saver is that lvm tools are so old that are before 8f518cf1979e4cbfce40f6ae1bed02bd5b9a5b35 commit that way there is only error but is not fatal. Lvm ignore mismach between transaction id from lvm and devicemapper but Debian users don't have so much luck.
@rhvgoyal

This comment has been minimized.

Show comment
Hide comment
@rhvgoyal

rhvgoyal Aug 18, 2015

Contributor

@cubbasa

On fedora we do lvextend all the time to grow pool dynamically while it is being filled up by docker. So I know it works fine.

docker-storage-setup by default creates an autoextension profile and attaches to docker pool volume so that auto extension can take place.

So I will recommend that ask debian maintainers to move to version -112 or higher and then test it all again.

Contributor

rhvgoyal commented Aug 18, 2015

@cubbasa

On fedora we do lvextend all the time to grow pool dynamically while it is being filled up by docker. So I know it works fine.

docker-storage-setup by default creates an autoextension profile and attaches to docker pool volume so that auto extension can take place.

So I will recommend that ask debian maintainers to move to version -112 or higher and then test it all again.

@cubbasa

This comment has been minimized.

Show comment
Hide comment
@cubbasa

cubbasa Aug 18, 2015

From what I've tested I get the same result. I could extend data pool on Ubuntu and beside the error it was working correctly. Ubuntu can't resize thin pool meta due old lvm tools. Actually I was looking for newer packages of lvm tools for Debian/Ubuntu but besides other people like me without result. Anyway really thank You for hint I will compile -112 myself and check again.

cubbasa commented Aug 18, 2015

From what I've tested I get the same result. I could extend data pool on Ubuntu and beside the error it was working correctly. Ubuntu can't resize thin pool meta due old lvm tools. Actually I was looking for newer packages of lvm tools for Debian/Ubuntu but besides other people like me without result. Anyway really thank You for hint I will compile -112 myself and check again.

@alexkuklin

This comment has been minimized.

Show comment
Hide comment
@alexkuklin

alexkuklin May 19, 2016

still the same:

docker-engine: 1.10.3-0~jessie
debian_version 8.3
fresh-installed server, after graceful reboot,

vgchange -a y
Thin pool transaction_id is 113, while expected 0.

I believe warning should be written somewhere about that, at least.

alexkuklin commented May 19, 2016

still the same:

docker-engine: 1.10.3-0~jessie
debian_version 8.3
fresh-installed server, after graceful reboot,

vgchange -a y
Thin pool transaction_id is 113, while expected 0.

I believe warning should be written somewhere about that, at least.

@antage

This comment has been minimized.

Show comment
Hide comment
@antage

antage Aug 8, 2016

On debian 8 thin pool can't be activated after reboot due to mismatched transaction id.
So using docker with devicemapper on dedicated thin pool is useless.

antage commented Aug 8, 2016

On debian 8 thin pool can't be activated after reboot due to mismatched transaction id.
So using docker with devicemapper on dedicated thin pool is useless.

@saytik

This comment has been minimized.

Show comment
Hide comment
@saytik

saytik Sep 8, 2016

the same problem on Debian 8

command: dpkg -l | grep lvm
ii liblvm2cmd2.02:amd64 2.02.111-2.2+deb8u1 amd64 LVM2 command library
ii lvm2 2.02.111-2.2+deb8u1 amd64 Linux Logical Volume Manager
command: dpkg -l | grep docker
ii docker-engine 1.12.1-0~jessie amd64 Docker: the open-source application container engine

After reboot thinpool volume is inactive ... and docker can't start.

command: vgchange -a y
1 logical volume(s) in volume group "vg1" now active
Thin pool transaction_id is 432, while expected 0.
3 logical volume(s) in volume group "vg0" now active
command: lvscan | grep pool
inactive '/dev/vg0/thinpool' [285.79 GiB] inherit

What can i do ?

saytik commented Sep 8, 2016

the same problem on Debian 8

command: dpkg -l | grep lvm
ii liblvm2cmd2.02:amd64 2.02.111-2.2+deb8u1 amd64 LVM2 command library
ii lvm2 2.02.111-2.2+deb8u1 amd64 Linux Logical Volume Manager
command: dpkg -l | grep docker
ii docker-engine 1.12.1-0~jessie amd64 Docker: the open-source application container engine

After reboot thinpool volume is inactive ... and docker can't start.

command: vgchange -a y
1 logical volume(s) in volume group "vg1" now active
Thin pool transaction_id is 432, while expected 0.
3 logical volume(s) in volume group "vg0" now active
command: lvscan | grep pool
inactive '/dev/vg0/thinpool' [285.79 GiB] inherit

What can i do ?

@thaJeztah thaJeztah added the kind/bug label Sep 14, 2016

@edwintorok

This comment has been minimized.

Show comment
Hide comment
@edwintorok

edwintorok Sep 19, 2016

I run into this problem every time I set up docker on a fresh machine or VM, e.g. just happened with 1.12.1-0~jessie: "Thin pool transaction_id is 4, while expected 0."

Workaround:
https://docs.docker.com/engine/userguide/storagedriver/device-mapper-driver/ used to have instructions that worked (by setting up separate data and metadata devices), so I go back to an older version http://web.archive.org/web/20151115000333/http://docs.docker.com/engine/userguide/storagedriver/device-mapper-driver/

I run into this problem every time I set up docker on a fresh machine or VM, e.g. just happened with 1.12.1-0~jessie: "Thin pool transaction_id is 4, while expected 0."

Workaround:
https://docs.docker.com/engine/userguide/storagedriver/device-mapper-driver/ used to have instructions that worked (by setting up separate data and metadata devices), so I go back to an older version http://web.archive.org/web/20151115000333/http://docs.docker.com/engine/userguide/storagedriver/device-mapper-driver/

@thaJeztah

This comment has been minimized.

Show comment
Hide comment
Member

thaJeztah commented Sep 19, 2016

ping @mbentley ^^

@mbentley

This comment has been minimized.

Show comment
Hide comment
@mbentley

mbentley Sep 19, 2016

Contributor

So yes, this appears to be directly related to the version of lvm2 in Debian Jessie (2.02.111-2.2+deb8u1). Grabbing lvm2=2.02.164-1 from testing (highly not suggested), it works fine. For users of Debian Jessie who must use device mapper, I'd stick to thick pools as @edwintorok mentioned. The actual link to the docs is: https://docs.docker.com/v1.10/engine/userguide/storagedriver/device-mapper-driver/#configure-direct-lvm-mode-for-production

When I updated the docs related to thin pools, I had only specifically tested with Debian Sid and RHEL 7.

Contributor

mbentley commented Sep 19, 2016

So yes, this appears to be directly related to the version of lvm2 in Debian Jessie (2.02.111-2.2+deb8u1). Grabbing lvm2=2.02.164-1 from testing (highly not suggested), it works fine. For users of Debian Jessie who must use device mapper, I'd stick to thick pools as @edwintorok mentioned. The actual link to the docs is: https://docs.docker.com/v1.10/engine/userguide/storagedriver/device-mapper-driver/#configure-direct-lvm-mode-for-production

When I updated the docs related to thin pools, I had only specifically tested with Debian Sid and RHEL 7.

@saytik

This comment has been minimized.

Show comment
Hide comment
@saytik

saytik Sep 26, 2016

confirm. I have updated LVM from testing repository on Debian 8 - fixed the problem.

saytik commented Sep 26, 2016

confirm. I have updated LVM from testing repository on Debian 8 - fixed the problem.

@HaiNguyen007

This comment has been minimized.

Show comment
Hide comment
@HaiNguyen007

HaiNguyen007 Sep 21, 2017

I also have updated LVM from testing repository on Debian 8 - fixed the problem.

I also have updated LVM from testing repository on Debian 8 - fixed the problem.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment