Package for pip, command line tool #37

slinnarsson · 2016-08-09T14:42:46Z

Create setup.py and register with Pypi to enable "pip install"

slinnarsson · 2016-08-09T21:21:13Z

Make it a command that can be run from the Terminal. The command would also have subcommands for managing loom files:

loom: start the server, launch user's default browser
loom put: upload a loom file to a remote server
loom clone: download a remote loom file to the local dataset folder
loom prepare: prepare a loom file for browsing (tSNE, 2D projection, etc.)
loom tsne: t-SNE projection
loom tile: prepare heatmap tile pyramid
loom stats: prepare standard stats (mean, cv etc)
loom make: Linnarsson lab-specific, make loom from SQL server

These commands are essentially the old pipeline but made available to command-line users. The previous "upload" functionality would be replaced by loom make, loom prepare and loom put.

This also means we need an endpoint to upload files, with proper authorization. Probably PUT /loom/{project}/{filename} makes sense.

JobLeonard · 2016-08-15T09:48:06Z

The build process doesn't quite get rid of old builds now:

creating build/bdist.linux-x86_64/egg/loompy/static/js
copying build/lib/loompy/static/js/bundle.0b80efca10453db2e7c3.js -> build/bdist.linux-x86_64/egg/loompy/static/js
copying build/lib/loompy/static/js/bundle.e4da1ada290b9b4911ed.js -> build/bdist.linux-x86_64/egg/loompy/static/js
copying build/lib/loompy/static/js/bundle.d54cd66797c031098437.js -> build/bdist.linux-x86_64/egg/loompy/static/js
copying build/lib/loompy/static/js/bundle.f43728d0560134400e61.js -> build/bdist.linux-x86_64/egg/loompy/static/js
copying build/lib/loompy/static/js/bundle.041e85a2226a2246bff2.js -> build/bdist.linux-x86_64/egg/loompy/static/js
copying build/lib/loompy/static/js/bundle.a1446e51ac3ed28e8b5f.js -> build/bdist.linux-x86_64/egg/loompy/static/js
copying build/lib/loompy/static/js/bundle.0eeff4f89ffabbcd28b5.js -> build/bdist.linux-x86_64/egg/loompy/static/js
copying build/lib/loompy/static/js/bundle.c48257cdf53e0d35b6cf.js -> build/bdist.linux-x86_64/egg/loompy/static/js
copying build/lib/loompy/static/js/bundle.855e53fee59303c0e234.js -> build/bdist.linux-x86_64/egg/loompy/static/js
copying build/lib/loompy/static/js/bundle.3701b14c83296ad01d6b.js -> build/bdist.linux-x86_64/egg/loompy/static/js
copying build/lib/loompy/static/js/bundle.45c52ee1902f254eaec5.js -> build/bdist.linux-x86_64/egg/loompy/static/js
copying build/lib/loompy/static/js/bundle.559e265394fb8c468593.js -> build/bdist.linux-x86_64/egg/loompy/static/js
copying build/lib/loompy/static/js/bundle.16240de0eac4ec259d8b.js -> build/bdist.linux-x86_64/egg/loompy/static/js
copying build/lib/loompy/static/js/bundle.287cac99b079aed08e34.js -> build/bdist.linux-x86_64/egg/loompy/static/js
copying build/lib/loompy/static/js/bundle.19c5bfe6824b2e7893b2.js -> build/bdist.linux-x86_64/egg/loompy/static/js
copying build/lib/loompy/static/js/bundle.1fdea5475c0bc1b1d87d.js -> build/bdist.linux-x86_64/egg/loompy/static/js
copying build/lib/loompy/static/js/bundle.a423a0bdbdca631fc243.js -> build/bdist.linux-x86_64/egg/loompy/static/js
copying build/lib/loompy/static/js/bundle.d20f5cbaede9d58efefe.js -> build/bdist.linux-x86_64/egg/loompy/static/js
copying build/lib/loompy/static/js/bundle.e3b07c47d0a8bc54ffc1.js -> build/bdist.linux-x86_64/egg/loompy/static/js
copying build/lib/loompy/static/js/bundle.1526b0744fe8bf4ea465.js -> build/bdist.linux-x86_64/egg/loompy/static/js
copying build/lib/loompy/static/js/bundle.0127b87ba1a9eff14af3.js -> build/bdist.linux-x86_64/egg/loompy/static/js
copying build/lib/loompy/static/js/bundle.feba36df0993cec39528.js -> build/bdist.linux-x86_64/egg/loompy/static/js
copying build/lib/loompy/static/js/bundle.f4729fc696ad9b540bfd.js -> build/bdist.linux-x86_64/egg/loompy/static/js
creating build/bdist.linux-x86_64/egg/loompy/static/css

I'm not sure if it's related, but I'm also having issues with the website loading older, cached versions of the JS bundle, which gets in the way debugging obviously.

JobLeonard · 2016-08-15T09:57:03Z

I tried python setup.py clean but that didn't do anything. Adding the following to the build file works though:

if [ -a ./python/build ]; then
  rm -r ./python/build
fi

JobLeonard · 2016-08-15T13:40:07Z

So I uncluttered the dataset-list and tried to upload, but then I got this message:

removing 'loompy-0.9.3' (and everything under it)
running upload
Traceback (most recent call last):
  File "setup.py", line 30, in <module>
    url = "https://github.com/linnarsson-lab/Loom",
  File "/home/job/Dev/anaconda2/lib/python2.7/distutils/core.py", line 151, in setup
    dist.run_commands()
  File "/home/job/Dev/anaconda2/lib/python2.7/distutils/dist.py", line 953, in run_commands
    self.run_command(cmd)
  File "/home/job/Dev/anaconda2/lib/python2.7/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File "/home/job/Dev/anaconda2/lib/python2.7/distutils/command/upload.py", line 60, in run
    self.upload_file(command, pyversion, filename)
  File "/home/job/Dev/anaconda2/lib/python2.7/distutils/command/upload.py", line 70, in upload_file
    raise AssertionError("unsupported schema " + schema)
AssertionError: unsupported schema

Since I didn't touch setup.py I'm not sure what the problem is. Anyway, I pushed the fix to git so you can upload to the server yourself for now. It defaults to genescape, since that one works properly right now.

slinnarsson · 2016-08-15T15:10:18Z

Ah. You need a .pypirc file with the correct credentials. See http://peterdowns.com/posts/first-time-with-pypi.html

I will give it to you separately so it doesn’t end up in the repository.

/Sten

Sten Linnarsson, PhD
Professor of Molecular Systems Biology
Karolinska Institutet
Unit of Molecular Neurobiology
Department of Medical Biochemistry and Biophysics
Scheeles väg 1, 171 77 Stockholm, Sweden
+46 8 52 48 75 77 (office)
+46 70 399 32 06 (mobile)

On 15 Aug 2016, at 15:40, Job van der Zwan <notifications@github.com mailto:notifications@github.com> wrote:

So I uncluttered the dataset-list and tried to upload, but then I got this message:

removing 'loompy-0.9.3' (and everything under it)
running upload
Traceback (most recent call last):
File "setup.py", line 30, in
url = "https://github.com/linnarsson-lab/Loom",
File "/home/job/Dev/anaconda2/lib/python2.7/distutils/core.py", line 151, in setup
dist.run_commands()
File "/home/job/Dev/anaconda2/lib/python2.7/distutils/dist.py", line 953, in run_commands
self.run_command(cmd)
File "/home/job/Dev/anaconda2/lib/python2.7/distutils/dist.py", line 972, in run_command
cmd_obj.run()
File "/home/job/Dev/anaconda2/lib/python2.7/distutils/command/upload.py", line 60, in run
self.upload_file(command, pyversion, filename)
File "/home/job/Dev/anaconda2/lib/python2.7/distutils/command/upload.py", line 70, in upload_file
raise AssertionError("unsupported schema " + schema)
AssertionError: unsupported schema

Since I didn't touch setup.py I'm not sure what the problem is. Anyway, I pushed the fix to git so you can upload to the server yourself for now. It defaults to genescape, since that one works properly right now.

—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHubhttps://github.com//issues/37#issuecomment-239804049, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AKKagx4aMGrMGGRIfoff3DgYAPr93ajtks5qgGw3gaJpZM4JgJxG.

slinnarsson · 2016-08-15T15:55:04Z

Ok clearly something is not working. I pull the latest version, build and check locally. I verify that I see the latest version with less clutter on the dataset view. Then I deploy, and confirm that the server indeed got upgraded to the latest version from PyPi. But still the server shows the previous version of the web page.

JobLeonard · 2016-08-15T18:14:39Z

Could it be that is has the same issue I had, with old .js files not being
cleaned?

On 15 August 2016 at 17:55, Sten Linnarsson notifications@github.com
wrote:

Ok clearly something is not working. I pull the latest version, build and
check locally. I verify that I see the latest version with less clutter on
the dataset view. Then I deploy, and confirm that the server indeed got
upgraded to the latest version from PyPi. But still the server shows the
previous version of the web page.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#37 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAP3AKQG9bbEKtm19d2RSBqzDbcoMFO8ks5qgIvYgaJpZM4JgJxG
.

JobLeonard · 2016-08-15T18:15:06Z

Although that wouldn't really make sense, given that I also updated the build script to fix that locally for me.

slinnarsson · 2016-08-18T15:44:58Z

Ok fixed it:

Reinstalled everything on the VM as user ubuntu (which is what the server says to use when I try to ssh as root).
Changed the deploy scripts to always use ubuntu@loom
Prevent loom-server from cacheing index.html and anything routed on client side
Uploaded new sample datasets with metadata
Upgraded to loompy 0.9.7 on the server
Pointed loom.linnarssonlab.org at the VM
Deleted the old containers

slinnarsson self-assigned this Aug 9, 2016

slinnarsson changed the title ~~Package for pip~~ Package for pip, command line tool Aug 10, 2016

slinnarsson closed this as completed Aug 18, 2016

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Package for pip, command line tool #37

Package for pip, command line tool #37

slinnarsson commented Aug 9, 2016

slinnarsson commented Aug 9, 2016 •

edited

Loading

JobLeonard commented Aug 15, 2016 •

edited

Loading

JobLeonard commented Aug 15, 2016

JobLeonard commented Aug 15, 2016

slinnarsson commented Aug 15, 2016

slinnarsson commented Aug 15, 2016

JobLeonard commented Aug 15, 2016

JobLeonard commented Aug 15, 2016

slinnarsson commented Aug 18, 2016

Package for pip, command line tool #37

Package for pip, command line tool #37

Comments

slinnarsson commented Aug 9, 2016

slinnarsson commented Aug 9, 2016 • edited Loading

JobLeonard commented Aug 15, 2016 • edited Loading

JobLeonard commented Aug 15, 2016

JobLeonard commented Aug 15, 2016

slinnarsson commented Aug 15, 2016

slinnarsson commented Aug 15, 2016

JobLeonard commented Aug 15, 2016

JobLeonard commented Aug 15, 2016

slinnarsson commented Aug 18, 2016

slinnarsson commented Aug 9, 2016 •

edited

Loading

JobLeonard commented Aug 15, 2016 •

edited

Loading