Adding new SystemD provisioner for VMware Photon OS.

[frapposelli]

This change adds a libmachine provisioner Photon OS and introduces an enhancement in utils.go for distros that have replaced netstat with ss (like Photon).

[frapposelli]

I see that the systemd provisioner was reverted, will do as well.

[nathanleclaire]

Yeah, keep an eye out on that one. I've submitted a PR to fix the issue why it was reverted.

[frapposelli]

We should be good with this, when the SystemD provisioner is added again I will revert to it.

[dgageot]

@frapposelli It's merged now!

[frapposelli]

you guys move too fast :-) reverted back to the systemd based provisioner

[frapposelli]

@dgageot is there a chance we can get this in for the 0.5.2 release? 😻

[dgageot]

@frapposelli what's the best way for me to test this?

[frapposelli]

@dgageot grab a mac and a copy of AppCatalyst (brew cask install vmware-appcatalyst) and use the AppCatalyst plugin available here: https://github.com/vmware/docker-machine-driver-vmwareappcatalyst

The plugin will use the embedded version of Photon OS that we ship with AppCatalyst.

[dgageot]

Btw, when it works, you might want to add your driver here

[dgageot]

@frapposelli I've just tried and failed. See vmware-archive/docker-machine-driver-vmwareappcatalyst#1

[dgageot]

Nice @frapposelli it works. I managed to provision a photon-based machine!

LGTM

[dgageot]

ping @nathanleclaire PTAL

[nathanleclaire]

We're interested, but need to ensure the version of Docker distributed with Photon matches with the officially distributed version.

On the release of Photon I've gotten, the checksums don't seem to match like they do with boot2docker. I've created a 1.8.1 boot2docker VM to verify this since that's the version in the Photon VM I got after a docker-machine create.

According to https://get.docker.com/builds/Linux/x86_64/docker-1.8.1.sha256, the SHA256 for the statically compiled and officially distributed Docker binary is:

843f90f5001e87d639df82441342e6d4c53886c65f72a5cc4765a7ba3ad4fc57

The sha256 digest in my Photon VM doesn't match for me, but it does for boot2docker:

$ docker-machine ssh 1.8.1 openssl dgst -sha256 /usr/local/bin/docker
SHA256(/usr/local/bin/docker)= 843f90f5001e87d639df82441342e6d4c53886c65f72a5cc4765a7ba3ad4fc57
$ docker-machine ssh photon openssl dgst -sha256 /usr/bin/docker-1.8.1
SHA256(/usr/bin/docker-1.8.1)= 45859761a5d34f9aeb425549b6a0ef7ab86f3343f733e766d27bd360e5068666

[frapposelli]

Understand the requirement, I'm pretty sure we just repackage the official binary from get.docker.com and the SPEC file we use for the docker package seems to confirm this https://github.com/vmware/photon/blob/release-1.0TP2/SPECS/docker/docker.spec

I'll take a look later today to see what's wrong with it.

[frapposelli]

@nathanleclaire I found the inconsistency, the build is exactly the same, the difference is that the binary we ship is stripped (most likely automatically done by the rpmbuild process):

photon [ ~ ]$ curl -L -O https://get.docker.com/builds/Linux/x86_64/docker-1.8.1
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 20.5M  100 20.5M    0     0  1160k      0  0:00:18  0:00:18 --:--:-- 1485k
photon [ ~ ]$ file docker-1.8.1
docker-1.8.1: ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), statically linked, for GNU/Linux 2.6.24, BuildID[sha1]=00677a720249ef2375c00e87ab8f00a67cf23bd7, not stripped
photon [ ~ ]$ file /usr/bin/docker-1.8.1
/usr/bin/docker-1.8.1: ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), statically linked, for GNU/Linux 2.6.24, BuildID[sha1]=8f4827f62919a94548d4b555868738b99de44b65, stripped

The build version shows the same version and build timestamp though (server is the engine shipped with photon):

photon [ ~ ]$ ./docker-1.8.1 version
Client:
 Version:      1.8.1
 API version:  1.20
 Go version:   go1.4.2
 Git commit:   d12ea79
 Built:        Thu Aug 13 02:49:29 UTC 2015
 OS/Arch:      linux/amd64

Server:
 Version:      1.8.1
 API version:  1.20
 Go version:   go1.4.2
 Git commit:   d12ea79
 Built:        Thu Aug 13 02:49:29 UTC 2015
 OS/Arch:      linux/amd64

We also updated the spec for 1.9.0, the package will probably be up in the repo quickly.

[frapposelli]

@nathanleclaire did you take a look at this? we're definitely sourcing the binary from get.docker.com and not building our own version.

[dgageot]

ping @nathanleclaire I LGTM already. I let you merge it

[dgageot]

@frapposelli Let's remove my LGTM for the moment.
What you are saying is that photon comes with docker pre-installed. This docker is an official version that is stripped.
Couldn't we let the provisioner install docker?

[frapposelli]

@dgageot docker is already installed in the base photon install, the provisioner can upgrade the package via tdnf (our package manager).

If you look at the SPEC file I linked above, we download from get.docker.com and package the binary in an RPM, at the end of the packaging process rpmbuild automatically strips the packages to remove debug symbols and reduce the size of the binary (that is common for many distros).

[frapposelli]

@dgageot @nathanleclaire can we agree on next steps?

I can't change what we already shipped in this version of AppCatalyst but I can change future versions, still it's nothing that is strictly related to docker-machine, I will work with the Photon team to make sure future packages are not stripped.

[nathanleclaire]

@frapposelli We are working with a variety of folks internally and externally to define what the expectations should be out of the provisioners (and Machine in general) and will be following up with some clearly defined and officially documented rules. They will be applied equally to all provisioners, including existing ones which were merged before they were defined (they will be removed if found to be in violation).

Until we get some clear rules out, we're going to have to hold on merging this. Sorry for the delay.

In terms of installation of Docker, we do know that we will be requiring all provisioners to ship officially distributed binaries and/or packages. Therefore, if you're going to install via RPM, we ask that you please use the officially distributed RPMs instead of packaging your own. The script at get.docker.com is probably the most canonical, definitive example of how to do this today. Take a look, for instance, at this section: https://github.com/docker/docker/blob/master/hack/install.sh#L383-L404

Thanks!

• N

[frapposelli]

@nathanleclaire thanks, really appreciate the clear response, will work with the photon guys to see if there are any challenges in using your RPM repo.

[GalGavu]

@frapposelli I tried your code and after I'm hitting into an issue when created vmwareappcatalyst driver based machine. I'm getting:

Error response from daemon: client is newer than server (client API version: 1.21, server API version: 1.20)

Do you have any advice?

[nathanleclaire]

@GalGavu Your Docker client is a newer version than the Docker daemon available. You should be able to use Docker by SSHing into the machine.

[dgageot]

@frapposelli @nathanleclaire I think we agree that we cannot merge this PR given that Docker should be installed by Machine, from official packages. Boot2docker being an exception that we fully control also.
Are you ok if the PR is closed until another version of Photon OS meets the requirements?

[nathanleclaire]

Are you ok if the PR is closed until another version of Photon OS meets the requirements?

I am OK with this -- maybe @frapposelli can give us an update on the progress. I would like to get some unambiguous documented guidelines in our repository as well.

[frapposelli]

Cool with it, quick update on the progress: I'm working on a PR for docker/docker to enable official builds for photon, once that is done I'll rework the PR to reflect.

[dgageot]

@frapposelli any update on the docker/docker PR?

[frapposelli]

@dgageot the PR (moby/moby#20108) was temporarily rejected until Photon reaches GA.

[dgageot]

@frapposelli thank you for the feedback! Since the underlying PR (moby/moby#20108) is closed, I'm going to close this PR too.
We will be pleased to merge a new PR when Photon reaches GA.
Thanks again.