Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ZFS Support #13

Open
au-phiware opened this issue Jan 11, 2017 · 70 comments

Comments

@au-phiware
Copy link

commented Jan 11, 2017

Upon executing docker run -e 'ACCEPT_EULA=Y' -e 'SA_PASSWORD=<YourStrong!Passw0rd>' -p 1433:1433 microsoft/mssql-server-linux, I encounter the following message:

Configuring Microsoft(R) SQL Server(R)...
Microsoft(R) SQL Server(R) setup failed with error code 1. Please check the setup log in /var/opt/mssql/log for more information.

The setup log contains:

This is an evaluation version.  There are [154] days left in the evaluation period.
2017-01-10 23:55:24.82 Server      Setup step is copying system data file 'C:\templatedata\master.mdf' to 'C:\var\opt\mssql\data\master.mdf'.
2017-01-10 23:55:24.90 Server      Setup step is copying system data file 'C:\templatedata\mastlog.ldf' to 'C:\var\opt\mssql\data\mastlog.ldf'.
2017-01-10 23:55:24.91 Server      Setup step is copying system data file 'C:\templatedata\model.mdf' to 'C:\var\opt\mssql\data\model.mdf'.
2017-01-10 23:55:24.92 Server      Setup step is copying system data file 'C:\templatedata\modellog.ldf' to 'C:\var\opt\mssql\data\modellog.ldf'.
2017-01-10 23:55:24.94 Server      Setup step is copying system data file 'C:\templatedata\msdbdata.mdf' to 'C:\var\opt\mssql\data\msdbdata.mdf'.
2017-01-10 23:55:24.95 Server      Setup step is copying system data file 'C:\templatedata\msdblog.ldf' to 'C:\var\opt\mssql\data\msdblog.ldf'.
2017-01-10 23:55:25.02 Server      Microsoft SQL Server vNext (CTP1.1) - 14.0.100.187 (X64) 
	Dec 10 2016 02:51:11 
	Copyright (C) 2016 Microsoft Corporation. All rights reserved.
	on Linux (Ubuntu 16.04.1 LTS)
2017-01-10 23:55:25.02 Server      UTC adjustment: 0:00
2017-01-10 23:55:25.02 Server      (c) Microsoft Corporation.
2017-01-10 23:55:25.03 Server      All rights reserved.
2017-01-10 23:55:25.03 Server      Server process ID is 4116.
2017-01-10 23:55:25.03 Server      Logging SQL Server messages in file 'C:\var\opt\mssql\log\errorlog'.
2017-01-10 23:55:25.03 Server      Registry startup parameters: 
	 -d C:\var\opt\mssql\data\master.mdf
	 -l C:\var\opt\mssql\data\mastlog.ldf
	 -e C:\var\opt\mssql\log\errorlog
2017-01-10 23:55:25.03 Server      Command Line Startup Parameters:
	 --setup
	 --sa-password
2017-01-10 23:55:25.04 Server      Error: 17113, Severity: 16, State: 1.
2017-01-10 23:55:25.04 Server      Error 87(The parameter is incorrect.) occurred while opening file 'C:\var\opt\mssql\data\master.mdf' to obtain configuration information at startup. An invalid startup option might have caused the error. Verify your startup options, and correct or remove them if necessary.
2017-01-10 23:55:25.05 Server      SQL Server shutdown has been initiated

The /var/opt/mssql/data directory contains:

-rw-r----- 1 root root  4194304 Jan 11 10:55 master.mdf
-rw-r----- 1 root root   524288 Jan 11 10:55 mastlog.ldf
-rw-r----- 1 root root  8388608 Jan 11 10:55 modellog.ldf
-rw-r----- 1 root root  8388608 Jan 11 10:55 model.mdf
-rw-r----- 1 root root 13959168 Jan 11 10:55 msdbdata.mdf
-rw-r----- 1 root root   524288 Jan 11 10:55 msdblog.ldf

Please advise.

@au-phiware

This comment has been minimized.

Copy link
Author

commented Jan 11, 2017

uname -a
Linux tecknack-corin 4.6.5-stable #54 SMP Tue Sep 13 11:33:24 AEST 2016 x86_64 Intel(R) Core(TM) i7-5500U CPU @ 2.40GHz GenuineIntel GNU/Linux
docker version
Client:
 Version:      1.12.1
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   23cf638
 Built:        
 OS/Arch:      linux/amd64

Server:
 Version:      1.12.1
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   23cf638
 Built:        
 OS/Arch:      linux/amd64
@twright-msft

This comment has been minimized.

Copy link
Collaborator

commented Jan 11, 2017

@au-phiware I tried running the exact command that you provided and it worked ok for me. In my case I was using Docker for Windows, but I've done essentially the same thing recently on RHEL and Docker for Mac. I'm wondering - is that the exact command you used or did you possibly try mounting a volume using docker run -v?

@au-phiware

This comment has been minimized.

Copy link
Author

commented Jan 11, 2017

Hi @twright-msft, you are right I did run it with -v initially (that's how I got the log contents) but before posting I made sure to run that exact command. I tried a number of variations, all had exactly the same output, and I'm assuming the setup log is the same too. So, I could be wrong, is it likely that the log would be different? I could copy the file out if you suspect there's something in that...?

@au-phiware

This comment has been minimized.

Copy link
Author

commented Jan 11, 2017

I have no doubt that there must be something particular about my kernel or filesystem. I use Gentoo and build my own kernel from git (the stable branch) but I'm not doing anything special there. I use zfs and so the docker storage driver is zfs too, could it be that there's an issue with accessing metadata or something else to do with the filesystem?

@au-phiware

This comment has been minimized.

Copy link
Author

commented Jan 11, 2017

Just to be sure I tried on my home machine (also Gentoo and ZFS) and used docker cp to retrieve the errorlog file. Here's what happened:

$ docker run -e 'ACCEPT_EULA=Y' -e 'SA_PASSWORD=<YourStrong!Passw0rd>' -p 1433:1433 microsoft/mssql-server-linux
Unable to find image 'microsoft/mssql-server-linux:latest' locally
latest: Pulling from microsoft/mssql-server-linux
aed15891ba52: Pull complete 
773ae8583d14: Pull complete 
d1d48771f782: Pull complete 
cd3d6cd6c0cf: Pull complete 
8ff6f8a9120c: Pull complete 
1fd7e8b10447: Pull complete 
bd485157db89: Pull complete 
273a1970ce9c: Pull complete 
d2d0b4dc209f: Pull complete 
Digest: sha256:f17b5a6d31f2e863581ba91169641a21f05584d9e5f82e43ee066c307e5550af
Status: Downloaded newer image for microsoft/mssql-server-linux:latest
Configuring Microsoft(R) SQL Server(R)...
Microsoft(R) SQL Server(R) setup failed with error code 1. Please check the setup log in /var/opt/mssql/log for more information.
$ docker ps -a | head -n 2
CONTAINER ID        IMAGE                               COMMAND                  CREATED             STATUS                      PORTS               NAMES
d66b9878ea41        microsoft/mssql-server-linux        "/bin/sh -c /opt/m..."   14 minutes ago      Exited (1) 14 minutes ago                       nervous_engelbart
$ docker cp d66b9878ea41:/var/opt/mssql/log/errorlog .
$ cat errorlog
ÿþ2017-01-11 20:59:36.77 Server      Microsoft SQL Server vNext (CTP1.1) - 14.0.100.187 (X64) 
	Dec 10 2016 02:51:11 
	Copyright (C) 2016 Microsoft Corporation. All rights reserved.
	on Linux (Ubuntu 16.04.1 LTS)
2017-01-11 20:59:36.78 Server      UTC adjustment: 0:00
2017-01-11 20:59:36.78 Server      (c) Microsoft Corporation.
2017-01-11 20:59:36.78 Server      All rights reserved.
2017-01-11 20:59:36.78 Server      Server process ID is 4116.
2017-01-11 20:59:36.78 Server      Logging SQL Server messages in file 'C:\var\opt\mssql\log\errorlog'.
2017-01-11 20:59:36.78 Server      Registry startup parameters: 
	 -d C:\var\opt\mssql\data\master.mdf
	 -l C:\var\opt\mssql\data\mastlog.ldf
	 -e C:\var\opt\mssql\log\errorlog
2017-01-11 20:59:36.78 Server      Command Line Startup Parameters:
	 --setup
	 --sa-password
2017-01-11 20:59:36.79 Server      Error: 17113, Severity: 16, State: 1.
2017-01-11 20:59:36.79 Server      Error 87(The parameter is incorrect.) occurred while opening file 'C:\var\opt\mssql\data\master.mdf' to obtain configuration information at startup. An invalid startup option might have caused the error. Verify your startup options, and correct or remove them if necessary.
2017-01-11 20:59:36.80 Server      SQL Server shutdown has been initiated
$ docker version
Client:
 Version:      1.13.0-rc3
 API version:  1.25
 Go version:   go1.7.1
 Git commit:   4d92237
 Built:        
 OS/Arch:      linux/amd64

Server:
 Version:      1.13.0-rc3
 API version:  1.25 (minimum version 1.12)
 Go version:   go1.7.1
 Git commit:   4d92237
 Built:        
 OS/Arch:      linux/amd64
 Experimental: false
$ uname -a
Linux fibonacci 4.3.6-stable #61 SMP Sun Dec 18 14:11:45 AEDT 2016 x86_64 Intel(R) Core(TM) i7-4710HQ CPU @ 2.50GHz GenuineIntel GNU/Linux
@twright-msft

This comment has been minimized.

Copy link
Collaborator

commented Jan 12, 2017

We havent done any testing on Gentoo/zfs yet. It's definitely possible there is an issue there with the kernel/file system.

@au-phiware

This comment has been minimized.

Copy link
Author

commented Jan 12, 2017

@Mobrockers

This comment has been minimized.

Copy link

commented Jan 25, 2017

I am having this exact same issue on Docker for windows.

@twright-msft

This comment has been minimized.

Copy link
Collaborator

commented Jan 26, 2017

One thing to check, especially if you are using Docker for Windows or Docker for Mac is that you adjust the amount of RAM that Docker Engine has available to it.
https://docs.docker.com/docker-for-mac/#/advanced
https://docs.docker.com/docker-for-windows/#/advanced

@Mobrockers

This comment has been minimized.

Copy link

commented Jan 26, 2017

The mssql image shows a clear and specific error message when docker engine has less than 4gb ram available. This is not the error I was getting. After a full docker reset the issue seems to have gone away now though.

@laurentleseigneur

This comment has been minimized.

Copy link

commented Feb 2, 2017

having same issue and error message using mac and docker v1.13 with 4gb memory

when mounting volume with -v option to bind /var/opt/mssql, we can see logs but .mdf id failing with error 87.

workaround is to start without this -v option, it works, but .mdf file are inside container

@twright-msft

This comment has been minimized.

Copy link
Collaborator

commented Feb 2, 2017

@laurentleseigneur This is a known issue with macOS and -v. We have a separate issue tracking that. #12
For now, you can use an attached Docker container volume for database file storage.
https://docs.docker.com/engine/tutorials/dockervolumes/#/creating-and-mounting-a-data-volume-container

@laurentleseigneur

This comment has been minimized.

Copy link

commented Feb 2, 2017

thanks @twright-msft for pointing this known issue, i will give a try to a docker compose using a container volume to store .mdf files

@sirio3mil

This comment has been minimized.

Copy link

commented Feb 13, 2017

I had the same error and I solved it by erasing the images and containers and re-creating them.

Docker version 1.13.1, build 092cba3
Windows 10 build 15031.rs2_release.170204-1546

@PatrickDePuydt

This comment has been minimized.

Copy link

commented Mar 8, 2017

@twright-msft Thanks for this thread; came here because I'm experiencing the same issue. I wanted to see if my case is a part of the known issue because I'm using mounted data volume containers.

In my docker-compose.yml I have a service that is configured:

  mssql:
    build: ./mssql
    env_file:
    - .env
    ports:
    - "1433:1433"
    volumes:
    - ./mssql/volumes/data:/var/opt/mssql

But when I pull up the logs from docker logs <container iD> It spits out:

2017-03-08 20:01:34.80 Server      Error: 17113, Severity: 16, State: 1.
2017-03-08 20:01:34.80 Server      Error 87(The parameter is incorrect.) occurred while opening file 'C:\var\opt\mssql\data\master.mdf' to obtain configuration information at startup. An invalid startup option might have caused the error. Verify your startup options, and correct or remove them if necessary.
2017-03-08 20:01:34.80 Server      SQL Server shutdown has been initiated

Is this part of the the known issue?

@twright-msft

This comment has been minimized.

Copy link
Collaborator

commented Mar 8, 2017

@PatrickDePuydt Are you on MacOS or Linux? If Linux, what distribution/version are you running on?

@PatrickDePuydt

This comment has been minimized.

Copy link

commented Mar 8, 2017

@twright-msft I'm running OS X Sierra but the base Docker image is a Centos7

@twright-msft

This comment has been minimized.

Copy link
Collaborator

commented Mar 8, 2017

@PatrickDePuydt OK, then yes, you are running into the issue I described above with using Docker for Mac and -v (volumes: in the way you are using it in docker-compose.yml is effectively docker run -v). Please use a 'data volume container' instead.
More info:
https://docs.docker.com/engine/tutorials/dockervolumes/#creating-and-mounting-a-data-volume-container

http://stackoverflow.com/questions/32908621/how-can-i-create-a-data-container-only-using-docker-compose-yml

@PatrickDePuydt

This comment has been minimized.

Copy link

commented Mar 8, 2017

@twright-msft Ok, thanks! Could you clear up one last thing? You mentioned using an attached Docker container volume for database file storage as a workaround, but isn't that what I'm doing in the volume declaration of the service: - ./mssql/volumes/data:/var/opt/mssql ?

@twright-msft

This comment has been minimized.

Copy link
Collaborator

commented Mar 8, 2017

As I understand it what you are doing there is mounting a directory on the host to a directory in the container. That's what doesnt work on macOS. What you want to do is mount a volume container to a directory in the mssql container.

@somashekarg

This comment has been minimized.

Copy link

commented Mar 11, 2017

This is an evaluation version. There are [156] days left in the evaluation period.
RegQueryValueEx HADR for key "Software\Microsoft\Microsoft SQL Server\MSSQL\MSSQLServer\HADR" failed.
2017-03-11 09:58:56.66 Server Microsoft SQL Server vNext (CTP1.3) - 14.0.304.138 (X64)
Feb 13 2017 16:49:12
Copyright (C) 2016 Microsoft Corporation. All rights reserved.
on Linux (Ubuntu 16.04.1 LTS)
2017-03-11 09:58:56.66 Server UTC adjustment: 0:00
2017-03-11 09:58:56.66 Server (c) Microsoft Corporation.
2017-03-11 09:58:56.66 Server All rights reserved.
2017-03-11 09:58:56.67 Server Server process ID is 4116.
2017-03-11 09:58:56.67 Server Logging SQL Server messages in file 'C:\var\opt\mssql\log\errorlog'.
2017-03-11 09:58:56.67 Server Registry startup parameters:
-d C:\var\opt\mssql\data\master.mdf
-l C:\var\opt\mssql\data\mastlog.ldf
-e C:\var\opt\mssql\log\errorlog
2017-03-11 09:58:56.69 Server Error: 17113, Severity: 16, State: 1.
2017-03-11 09:58:56.69 Server Error 3(The system cannot find the path specified.) occurred while opening file 'C:\var\opt\mssql\data\master.mdf' to obtain configuration information at startup. An invalid startup option might have caused the error. Verify your startup options, and correct or remove them if necessary.
2017-03-11 09:58:56.70 Server SQL Server shutdown has been initiated

I have the same problem but not in docker. it running and working in docker after pushing the docker image to heroku app it is giving the above message message. anybody has any idea about this issue.
Please help if anybody knows

Thanks

@au-phiware

This comment has been minimized.

Copy link
Author

commented May 15, 2017

(I feels as if this thread has been hijacked by Mac users)

I decided to take ZFS out of the mix:

  1. First I confirmed I was still receiving the error with the latest image.
  2. I create a zvol and formatted it with ext4 and mounted it as my /var/lib/docker directory.
  3. Started the docker daemon and ensured it wasn't using zfs as the storage device.
  4. Tried again.

This time I didn't receive the error, the container stayed up and I have an operational mssql server on linux!

I guess the next question is: where does the fault lie? This project (seems unlikely)? Docker (more specifically the zfs storage driver)? Or ZFS?

I think I'll close this issue and go ask Docker.

For the record here's my working configuration:

$ docker info
Containers: 1
 Running: 1
 Paused: 0
 Stopped: 0
Images: 2
Server Version: 1.12.1
Storage Driver: devicemapper
 Pool Name: docker-230:1-131073-pool
 Pool Blocksize: 65.54 kB
 Base Device Size: 10.74 GB
 Backing Filesystem: ext4
 Data file: /dev/loop0
 Metadata file: /dev/loop1
 Data Space Used: 2.485 GB
 Data Space Total: 107.4 GB
 Data Space Available: 7.923 GB
 Metadata Space Used: 2.257 MB
 Metadata Space Total: 2.147 GB
 Metadata Space Available: 2.145 GB
 Thin Pool Minimum Free Space: 10.74 GB
 Udev Sync Supported: true
 Deferred Removal Enabled: false
 Deferred Deletion Enabled: false
 Deferred Deleted Device Count: 0
 Data loop file: /var/lib/docker/devicemapper/devicemapper/data
 WARNING: Usage of loopback devices is strongly discouraged for production use. Use `--storage-opt dm.thinpooldev` to specify a custom block storage device.
 Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
 Library Version: 1.02.93 (2015-01-30)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: seccomp
Kernel Version: 4.6.5-stable
Operating System: Gentoo/Linux
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 15.59 GiB
Name: tecknack-corin
ID: JXJV:BP5E:T547:PZTF:4JSO:6SV5:B7EO:XA3U:EPV7:7Z5H:BS2Y:SAV7
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Insecure Registries:
 127.0.0.0/8
@gigatexal

This comment has been minimized.

Copy link

commented May 15, 2017

@Mic92

This comment has been minimized.

Copy link

commented May 15, 2017

Here is a full syscall trace from machine with zfsonlinux:

Steps to reproduce

  1. docker run -ti -e 'ACCEPT_EULA=Y' -e 'SA_PASSWORD=<YourStrong!Passw0rd>' -p 1433:1433 microsoft/mssql-server-linux bash
  2. root@f5f756e8d68c:/# apt update && apt install strace
  3. strace -f -o /tmp/sqlservr.log /opt/mssql/bin/sqlservr

https://dl.thalheim.io/Z03MnoPGwbCXhJOXpBvgwg/sqlservr.log?dl=1

I tried to reason about the repeated sequence of syscalls (some strange enterprise retry handler), but found not failing syscall. Without access to the source code and the uninformative error message, I have no clue what's going on here.

@twright-msft

This comment has been minimized.

Copy link
Collaborator

commented May 15, 2017

FWIW we have done zero testing on zfs. @au-phiware Please let us know the issue ID if you do raise this with Docker in a way that is publicly trackable. Thanks!

@au-phiware

This comment has been minimized.

Copy link
Author

commented May 15, 2017

The issue can be viewed at moby/moby#33191

@Mic92

This comment has been minimized.

Copy link

commented May 16, 2017

@au-phiware I do not think docker is the best issue tracker for this, as we miss domain knowledge about mssql internals.

@au-phiware

This comment has been minimized.

Copy link
Author

commented May 16, 2017

@Mic92 yes, well I'm really not sure where to go... I'll try recreate the issue with docker removed from the mix.

Is there a better issue tracker someplace or should I reopen this one?

@Mic92

This comment has been minimized.

Copy link

commented Feb 16, 2018

In regards to O_DIRECT I don't see other drawbacks. I have not tested mssql intensively to see what it does elsewhere.

@t-oster

This comment has been minimized.

Copy link

commented Feb 21, 2018

Thank you @Mic92 it seems to work very good, maybe I will switch our production system from ext4/zvol directly to zfs (after some tests). If anyone else needs this, here you are: https://github.com/t-oster/mssql-docker-zfs

@Spongman

This comment has been minimized.

Copy link

commented Feb 22, 2018

i just ran into this today. IMO this is more than an enhancement - zfs is the default storage driver for docker on systems that support it.

@Smithx10

This comment has been minimized.

Copy link

commented Mar 25, 2018

Is mssql going to implement something similar to the innodb_flush_method?

It allows for the following types of methods to flush writes to disk:
fsync
O_DSYNC
littlesync
nosync
O_DIRECT
O_DIRECT_NO_FSYNC

Is there any news on this?

@twright-msft

This comment has been minimized.

Copy link
Collaborator

commented Mar 26, 2018

@Smithx10 - We don’t currently have any plans to support additional/configurable flush methods. Do you think we need to?

@Smithx10

This comment has been minimized.

Copy link

commented Mar 26, 2018

@twright-msft , Is there a plan to support ZFS and other operating systems without forcing them to hack around the real problem using LD_PRELOAD?

To be honest, it would be awesome if there was native FreeBSD and Illumos support. My experience would tell me that... this probably will never happen, but hey... It's a New Microsoft right?

:(

DirectIO via the O_DIRECT flag was originally introduced in XFS by IRIX
for database workloads. Its purpose was to allow the database to bypass
the page and buffer caches to prevent unnecessary IO operations (e.g.
readahead) while preventing contention for system memory between the
database and kernel caches.

Unfortunately, the semantics were never defined in any standard. The
semantics of O_DIRECT in XFS in Linux are as follows:

  1. O_DIRECT requires IOs be aligned to backing device's sector size.
  2. O_DIRECT performs unbuffered IO operations between user memory and block
    device (DMA when the block device is physical hardware).
  3. O_DIRECT implies O_DSYNC.
  4. O_DIRECT disables any locking that would serialize IO operations.

The first is not possible in ZFS beause there is no backing device in
the general case.

The second is not possible in ZFS in the presence of compression because
that prevents us from doing DMA from user pages. If we relax the
requirement in the case of compression, we encunter another hurdle. In
specific, avoiding the userland to kernel copy risks other userland
threads modifying buffers during compression and checksum computations.
For compressed data, this would cause undefined behavior while for
checksums, this would imply we write incorrect checksums to disk. It
would be possible to avoid those issues if we modify the page tables to
make any changes by userland to memory trigger page faults and perform
CoW operations. However, it is unclear if it is wise for a filesystem
driver to do this.

The third is doable, but we would need to make ZIL perform indirect
logging to avoid writing the data twice.

The fourth is already done for all IO in ZFS.

Other Linux filesystems such as ext4 do not follow #3. Mac OS X does not
implement O_DIRECT, but it does implement F_NOCACHE, which is similiar
to #2 in that it prevents new data from being cached. AIX relaxes #3 by
only committing the file data to disk. Metadata updates required should
the operations make the file larger are asynchronous unless O_DSYNC is
specified.

On Solaris and Illumos, there is a library function called directio(3C)
that allows userspace to provide a hint to the filesystem that DirectIO
is useful, but the filesystem is free to ignore it. The semantics are
also entirely a filesystem decision. Those that do not implement it
return ENOTTY.

Given the lack of standardization and ZFS' heritage, one solution to
provide compatibility with userland processes that expect DirectIO is to
treat DirectIO as a hint that we ignore. This can be done trivially by
implementing a shim that maps aops->direct_IO to AIO. There is also
already code in ZoL for bypassing the page cache when O_DIRECT is
specified, but it has been inert until now.

If it turns out that it is acceptable for a filesystem driver to
interact with the page tables, the scatter-gather list work will need be
finished and we would need to utilize the page tables to make operations
on the userland pages safe.

References:
http://xfs.org/docs/xfsdocs-xml-dev/XFS_User_Guide/tmp/en-US/html/ch02s09.html
https://blogs.oracle.com/roch/entry/zfs_and_directio
https://ext4.wiki.kernel.org/index.php/Clarifying_Direct_IO's_Semantics
https://illumos.org/man/3c/directio
https://developer.apple.com/library/mac/#documentation/Darwin/Reference/ManPages/man2/fcntl.2.html
https://lists.apple.com/archives/filesystem-dev/2007/Sep/msg00010.html

:(

Using O_DIRECT well requires taking on a surprising amount of additional resource management responsibility. Also, the implementation is not portable and, combined with the additional responsibility previously mentioned, adds a ton of conditional compilation. The alignment requirements are not an issue at all; anyone using O_DIRECT is also going to be used properly aligned I/O buffers in any case.O_DIRECT can have a lot of benefits. However, while O_DIRECT may be simple to use by itself, the other design and implementation requirements that are indirectly dragged along are not trivial by any means. Consequently, you should not be using O_DIRECT unless you can manage the substantial indirect overhead in implementation. It turns off a lot of OS features most developers take for granted. | Using O_DIRECT well requires taking on a surprising amount of additional resource management responsibility. Also, the implementation is not portable and, combined with the additional responsibility previously mentioned, adds a ton of conditional compilation. The alignment requirements are not an issue at all; anyone using O_DIRECT is also going to be used properly aligned I/O buffers in any case.O_DIRECT can have a lot of benefits. However, while O_DIRECT may be simple to use by itself, the other design and implementation requirements that are indirectly dragged along are not trivial by any means. Consequently, you should not be using O_DIRECT unless you can manage the substantial indirect overhead in implementation. It turns off a lot of OS features most developers take for granted.
Using O_DIRECT well requires taking on a surprising amount of additional resource management responsibility. Also, the implementation is not portable and, combined with the additional responsibility previously mentioned, adds a ton of conditional compilation. The alignment requirements are not an issue at all; anyone using O_DIRECT is also going to be used properly aligned I/O buffers in any case.O_DIRECT can have a lot of benefits. However, while O_DIRECT may be simple to use by itself, the other design and implementation requirements that are indirectly dragged along are not trivial by any means. Consequently, you should not be using O_DIRECT unless you can manage the substantial indirect overhead in implementation. It turns off a lot of OS features most developers take for granted.
 

:(

  1. Not that it's particularly bad for PostgreSQL, but as a point of possible future interest, I found out that the Linux kernel does not (and most likely will not in the future) recommend mixing O_DIRECT and non-O_DIRECT I/O on a single file. Doing this is expected to result in very poor performance, because the use of O_DIRECT causes page cache invalidations. I haven't been able to think of a scenario in which this will actually hurt PostgreSQL users today, because we only use O_DIRECT for WAL, and then only if wal_sync_method = open_sync or open_datasync, and then only neither archiving nor streaming replication is in use. However, it's certainly worth keeping in mind if we ever consider expanding the use of O_DIRECT.

:(

Richrd Yao says:
July 7, 2013 at 8:56 am
Vadim, O_DIRECT was designed for in-place filesystems to allow IO to bypass the filesystem layer and caching. A literal implementation of O_DIRECT in a copy-on-write filesystem like ZFS is not possible (because checksum and parity calculations must be done). It is possible to implement it by effectively ignoring the O_DIRECT flag, but I imagine that would defeat the purpose. I imagine is the main reason the solution where O_DIRECT is ignored has not been implemented is that Linux uses a different code path for O_DIRECT and time spent implementing the separate code path is time that could be spent on other bugs.

Most of the development of ZFSOnLinux over the past two years has focused on making it ready for the first stable release. Adding O_DIRECT support did not help contribute to that, so it was a low priority. I imagine that adding O_DIRECT support would occur rather quickly if someone were to write a patch to add it that works. However, adding it would be misleading unless O_DIRECT is implemented in a way that provided some kind of tangible benefit over not using it.

Lastly, ZFSOnLinux development is done by a few professional developers at LLNL and volunteers, such as myself, that happen to use it. LLNL uses ZFSOnLinux as an OSD for the Lustre filesystem on the Sequoia suprecomputer while volunteers tend to use it on either servers or desktops. O_DIRECT is currently scheduled for a release rather far in the future because none of us have any need for O_DIRECT. It should be possible to configure your software to not use O_DIRECT, so doing it sooner does not seem like it should be a priority.

@Mic92

This comment has been minimized.

Copy link

commented Mar 28, 2018

But hey there is still postgresql/mariadb which will perform faster anyway on *bsd/linux.

@Smithx10

This comment has been minimized.

Copy link

commented Mar 28, 2018

@Mic92 Sometimes as an operator, I get in situations where .net developers want to use mssql, or the Enterprise needs that Windows Stamp.

@Mic92

This comment has been minimized.

Copy link

commented Mar 28, 2018

Well for professional usage you don't want to use the linux port anyway... It does so many strange things I have seen in strace.

@twright-msft

This comment has been minimized.

Copy link
Collaborator

commented Mar 30, 2018

Hmmm.. that URL isnt resolving for me. Can you please check?

@twright-msft

This comment has been minimized.

Copy link
Collaborator

commented Mar 31, 2018

Thanks for the heads up @Smithx10 - we are taking a look at it.

@tisoft

This comment has been minimized.

Copy link

commented Sep 10, 2018

Starting with https://github.com/zfsonlinux/zfs/releases/tag/zfs-0.8.0-rc1 ZFS on Linux supports Direct IO. A quick test with the mssql image succeeded for me. Can't say anything about performance/stability though.

@t-oster

This comment has been minimized.

Copy link

commented Sep 10, 2018

That's good news! I am successfully running some mssql in docker on ZFS with the LD_PRELOAD hack and did not have any problems so far. But it's better if that is not necessary anymore. I will report as soon as this version is on archzfs, so I can test.

@fuuqiu

This comment has been minimized.

Copy link

commented Oct 6, 2018

看了一遍都是英文的,反正英文也不好.就直接上解决思路吧,出现这样的问题呢是因为启动的时候使用了-V挂载数据卷,因为也只是为了有个mssql服务器来调试环境,也就懒得去折腾数据是在本地还是容器中,直接去掉-V就开工干活了,问题也解决了

@donhuvy

This comment has been minimized.

Copy link

commented Nov 24, 2018

I am using macOS Mojave 10.14.1

➜  ~ docker run -e 'ACCEPT_EULA=Y' -e 'SA_PASSWORD=123456a@' -p 1433:1433 -d -v /Users/donhuvy/Documents/mssql:/var/opt/mssql -d microsoft/mssql-server-linux:2017-CU8
9d4bd2a1da7a3bece6ce1052bc221538fea44b0469c19543e670b2cf5f2a6205
➜  ~ docker logs -f 9d4bd2a1da7a3bece6ce1052bc221538fea44b0469c19543e670b2cf5f2a6205
2018-11-24 07:16:46.99 Server      Microsoft SQL Server 2017 (RTM-CU8) (KB4338363) - 14.0.3029.16 (X64) 
	Jun 13 2018 13:35:56 
	Copyright (C) 2017 Microsoft Corporation
	Developer Edition (64-bit) on Linux (Ubuntu 16.04.4 LTS)
2018-11-24 07:16:47.01 Server      UTC adjustment: 0:00
2018-11-24 07:16:47.01 Server      (c) Microsoft Corporation.
2018-11-24 07:16:47.01 Server      All rights reserved.
2018-11-24 07:16:47.01 Server      Server process ID is 4120.
2018-11-24 07:16:47.01 Server      Logging SQL Server messages in file '/var/opt/mssql/log/errorlog'.
2018-11-24 07:16:47.01 Server      Registry startup parameters: 
	 -d /var/opt/mssql/data/master.mdf
	 -l /var/opt/mssql/data/mastlog.ldf
	 -e /var/opt/mssql/log/errorlog
2018-11-24 07:16:47.04 Server      Error: 17113, Severity: 16, State: 1.
2018-11-24 07:16:47.04 Server      Error 87(The parameter is incorrect.) occurred while opening file '/var/opt/mssql/data/master.mdf' to obtain configuration information at startup. An invalid startup option might have caused the error. Verify your startup options, and correct or remove them if necessary.
➜  ~ 
@twright-msft

This comment has been minimized.

Copy link
Collaborator

commented Nov 25, 2018

@donhuvy - macos volume mounting is not currently working/supported. Please see the following issue tracking adding support for this in the future:
#12

@donhuvy

This comment has been minimized.

Copy link

commented Nov 30, 2018

@twright-msft Thank you for your explaination! I have a question (Outside the issue). If I run command

docker run -e 'ACCEPT_EULA=Y' -e 'SA_PASSWORD=123456a@' -p 1433:1433 -d microsoft/mssql-server-linux:2017-CU8

When I quite Docker, my database schema is removed. Then I must run creating new schema SQL script again. When I turn off my computer, then I turn on. I must create database schema(s) again. I mean that, I don't know how to save state of existing database schema(s) after quitting Docker.

(Because I don't know much about Docker) Please help me, show me command, how to saving database schema(s) (even when I turn off computer then turn on to continuing my old work)?

@sirio3mil

This comment has been minimized.

Copy link

commented Nov 30, 2018

@donhuvy check this basically you have two options, a docker volumes or bind mount a local folder, by default docker doesn't create this.

@twright-msft

This comment has been minimized.

Copy link
Collaborator

commented Dec 1, 2018

Yep, what @sirio3mil linked to is the general concept. Here's some details and examples for SQL Server specifically. https://docs.microsoft.com/en-us/sql/linux/sql-server-linux-configure-docker?view=sql-server-2017#persist

Just a reminder that -v mounting to a host file system ("bind mount") doesnt work on macos (#12 ). If you are using macos you'll need to use the mounted docker volume approach (docker volume create and then mount the docker volume to your container when you run it.)

@silverspringhusker

This comment has been minimized.

Copy link

commented Feb 6, 2019

Starting with https://github.com/zfsonlinux/zfs/releases/tag/zfs-0.8.0-rc1 ZFS on Linux supports Direct IO. A quick test with the mssql image succeeded for me. Can't say anything about performance/stability though.

That's good. Every few months, I set up a new server and forget that I have to put my SQL Server on non-ZFS volumes. On first start up, I scratch my head for an hour trying to figure it out. Hopefully this fixes the problem.
But until ZFS 0.8 is in production and widely available, if MS fixes it on their end, would that resolve the issue for MacOS too?

@dropsonic

This comment has been minimized.

Copy link

commented Mar 11, 2019

I have exactly the same issue when using docker-compose up with Docker for Windows.

@eggeggss

This comment has been minimized.

Copy link

commented Mar 21, 2019

看了一遍都是英文的,反正英文也不好.就直接上解决思路吧,出现这样的问题呢是因为启动的时候使用了-V挂载数据卷,因为也只是为了有个mssql服务器来调试环境,也就懒得去折腾数据是在本地还是容器中,直接去掉-V就开工干活了,问题也解决了

Do not hang volume? Then your docker restart data still?

@t-oster

This comment has been minimized.

Copy link

commented May 28, 2019

ZFS 0.8 is released and included at least in ArchLinux. I tested it and it works, so this can be closed IMHO. If you are stuck with an old version, use https://github.com/t-oster/mssql-docker-zfs

@Y-YJ

This comment has been minimized.

Copy link

commented Jul 15, 2019

原来的

volumes:
- ../data/mssql:/var/opt/mssql

改成

volumes:
- ../data/mssql:/var/opt/mssql
- /var/opt/mssql/data

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
You can’t perform that action at this time.