New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add snapcraft packaging #4852

Merged
merged 2 commits into from Mar 23, 2017

Conversation

Projects
None yet
2 participants
@mhall119
Contributor

mhall119 commented Feb 1, 2017

Adds snapcraft build configuration and command to run an in-snap Python 2 runtime. This allows MXNet to be used either by calling "/snap/bin/mxnet.python" or by adding the following to your environment and using your system's python:

LD_LIBRARY_PATH=/snap/mxnet/current/lib:/snap/mxnet/current/usr/lib:$LD_LIBRARY_PATH
PYTHONPATH=/snap/mxnet/current/lib/python2.7/site-packages/:/snap/mxnet/current/usr/lib/python2.7/dist-packages/:$PYTHONPATH

The snap can be build with "snapcraft snap" on Ubuntu 16.04 or later. Use "snapcraft cleanbuild" to build it in an LXC container instead.

@piiswrong

This comment has been minimized.

Show comment
Hide comment
@piiswrong

piiswrong Feb 1, 2017

Contributor

distribution is here https://github.com/dmlc/mxnet-distro
It currently publsh to pypi. You can make it publish to snap also.

The benefit of using snap is you can also install scala and R packages. Could you add that?

Contributor

piiswrong commented Feb 1, 2017

distribution is here https://github.com/dmlc/mxnet-distro
It currently publsh to pypi. You can make it publish to snap also.

The benefit of using snap is you can also install scala and R packages. Could you add that?

@piiswrong

This comment has been minimized.

Show comment
Hide comment
@piiswrong

piiswrong Feb 1, 2017

Contributor

Also would be great if there is a gpu enabled option.

and we want to enable USE_MKL2017 for cpu only build

Contributor

piiswrong commented Feb 1, 2017

Also would be great if there is a gpu enabled option.

and we want to enable USE_MKL2017 for cpu only build

@mhall119

This comment has been minimized.

Show comment
Hide comment
@mhall119

mhall119 Mar 8, 2017

Contributor

@piiswrong I've been looking at that mxnet-distro repo, trying to understand where and how you want the snapcraft integrated. Do you simply want the files in this PR submitted there (with modification to paths) or do you want it to be fully integrated into the Travis configuration so that it will create and publish the snap after the Travis build (a more involved process than this PR)?

Contributor

mhall119 commented Mar 8, 2017

@piiswrong I've been looking at that mxnet-distro repo, trying to understand where and how you want the snapcraft integrated. Do you simply want the files in this PR submitted there (with modification to paths) or do you want it to be fully integrated into the Travis configuration so that it will create and publish the snap after the Travis build (a more involved process than this PR)?

@piiswrong

This comment has been minimized.

Show comment
Hide comment
@piiswrong

piiswrong Mar 8, 2017

Contributor

Can one of the admins verify this patch?

Contributor

piiswrong commented Mar 8, 2017

Can one of the admins verify this patch?

@piiswrong

This comment has been minimized.

Show comment
Hide comment
@piiswrong

piiswrong Mar 8, 2017

Contributor

It depends on how you want to do the releasing. We can decide that later.
First we should figure out how to support scala, R, gpu, mkl, etc. Can snap detect gpu?

Contributor

piiswrong commented Mar 8, 2017

It depends on how you want to do the releasing. We can decide that later.
First we should figure out how to support scala, R, gpu, mkl, etc. Can snap detect gpu?

@mhall119

This comment has been minimized.

Show comment
Hide comment
@mhall119

mhall119 Mar 10, 2017

Contributor

Scala and R can either be bundled into the same .snap as the python library, or we can split this into 3 different language-specific snaps, depending on how desirable small packages are and how common it is for a user to need support for more than one language.

My understanding is that mxnet can be built to use GPU if the build host supports it. Because the use-case for this snap will have the code being executed normally (i.e. without a sandbox) there should be nothing stopping it from detecting and using a GPU at runtime.

Contributor

mhall119 commented Mar 10, 2017

Scala and R can either be bundled into the same .snap as the python library, or we can split this into 3 different language-specific snaps, depending on how desirable small packages are and how common it is for a user to need support for more than one language.

My understanding is that mxnet can be built to use GPU if the build host supports it. Because the use-case for this snap will have the code being executed normally (i.e. without a sandbox) there should be nothing stopping it from detecting and using a GPU at runtime.

@mhall119

This comment has been minimized.

Show comment
Hide comment
@mhall119

mhall119 Mar 23, 2017

Contributor

@piiswrong what should I do to keep moving this PR along? I'm unclear from the last comments if you are waiting on me or someone else.

Contributor

mhall119 commented Mar 23, 2017

@piiswrong what should I do to keep moving this PR along? I'm unclear from the last comments if you are waiting on me or someone else.

@piiswrong

This comment has been minimized.

Show comment
Hide comment
@piiswrong

piiswrong Mar 23, 2017

Contributor

I can merge this right now but what we really want is binaries on the server now config file in the repo.
I'm just trying to figure out how that's going to happen

Contributor

piiswrong commented Mar 23, 2017

I can merge this right now but what we really want is binaries on the server now config file in the repo.
I'm just trying to figure out how that's going to happen

@piiswrong

This comment has been minimized.

Show comment
Hide comment
@piiswrong

piiswrong Mar 23, 2017

Contributor

On detecting gpu, we will build separate binaries for cpu/gpu. Can snap figure out which binary to download based on whether the system has gpu?

Contributor

piiswrong commented Mar 23, 2017

On detecting gpu, we will build separate binaries for cpu/gpu. Can snap figure out which binary to download based on whether the system has gpu?

@piiswrong

This comment has been minimized.

Show comment
Hide comment
@piiswrong

piiswrong Mar 23, 2017

Contributor

Anyway I can merge this first if you need it merged to take the next steps.

Contributor

piiswrong commented Mar 23, 2017

Anyway I can merge this first if you need it merged to take the next steps.

@piiswrong piiswrong merged commit 8f8d834 into apache:master Mar 23, 2017

6 of 9 checks passed

continuous-integration/jenkins/pr-merge The build of this commit was aborted
Details
continuous-integration/travis-ci/pr The Travis CI build failed
Details
mxnet-pr-cpu-mkl
Details
Sanity Check Build finished.
Details
continuous-integration/appveyor/pr AppVeyor build succeeded
Details
default Build finished.
Details
mxnet-pr-cpu
Details
mxnet-pr-gpu
Details
mxnet-pr-r
Details
@mhall119

This comment has been minimized.

Show comment
Hide comment
@mhall119

mhall119 Mar 23, 2017

Contributor

Do you mean binary snap packages? If so, you can use https://build.snapcraft.io to create them, triggered by commits here. It's like Travis, only made specifically for creating and publishing snaps.

The "snap install" command doesn't currently give any information about GPU. We could have two separate packages, or include both sets of binaries in the same package

Contributor

mhall119 commented Mar 23, 2017

Do you mean binary snap packages? If so, you can use https://build.snapcraft.io to create them, triggered by commits here. It's like Travis, only made specifically for creating and publishing snaps.

The "snap install" command doesn't currently give any information about GPU. We could have two separate packages, or include both sets of binaries in the same package

@piiswrong

This comment has been minimized.

Show comment
Hide comment
@piiswrong

piiswrong Mar 23, 2017

Contributor

I see. Then I think we should move these files to mxnet-distro which is updated when ever there is a new release.
How do we configure it to build multiple binaries?

Contributor

piiswrong commented Mar 23, 2017

I see. Then I think we should move these files to mxnet-distro which is updated when ever there is a new release.
How do we configure it to build multiple binaries?

@mhall119

This comment has been minimized.

Show comment
Hide comment
@mhall119

mhall119 Mar 23, 2017

Contributor

build.snapcraft.io will default to building on multiple arches (x86_64, armhf and arm64 I think). It will publish them to an "edge" channel in the store that is the equivalent of daily builds, and only seen by people who specifically ask for the bleeding edge packages. They can then be tested and promoted into "beta" and "stable" channels. By default users will only see the versions published into "stable"

Contributor

mhall119 commented Mar 23, 2017

build.snapcraft.io will default to building on multiple arches (x86_64, armhf and arm64 I think). It will publish them to an "edge" channel in the store that is the equivalent of daily builds, and only seen by people who specifically ask for the bleeding edge packages. They can then be tested and promoted into "beta" and "stable" channels. By default users will only see the versions published into "stable"

ZihengJiang added a commit to ZihengJiang/mxnet that referenced this pull request Apr 4, 2017

Add snapcraft packaging (#4852)
* Add snapcraft.yaml

* Add in-snap python interpreter

Guneet-Dhillon pushed a commit to Guneet-Dhillon/mxnet that referenced this pull request Sep 13, 2017

Add snapcraft packaging (#4852)
* Add snapcraft.yaml

* Add in-snap python interpreter
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment