A tool for running bioinformatics workflows locally or in the cloud.
Clone or download
nhammond Merge 0.7.1 from Development (#563)
* roll back untested log cleanup feature

* fix bug that masked errors in failed docker pull attempts by taskmonitor

* delete cmd for runs, templates, files

* run deletion handles datanodes, children

* object deletion from cmdline client

* cleanup runs with invalid user_inputs

* file export handles metadata

* file import handles metadata

* track linked files in database

* refactor FileManager into Export/ImportManager

* unit tests for loomengine_utils.connection

* wildcards for google storage, unit tests for file_utils

* loomengine_utils unittests

* template bulk export

* template cleans up DataNodes on delete

* migrate template import to utils

* ensure DataNode children are sorted by index

* import templates with dependencies

* bulk-import templates

* bulk-import runs

* attach nested objects on run import

* finish removing django-mptt

* optimize deserialization

* update template and run export

* update loomengine_utils unit tests

* push outputs before setting run status to finished

* bugfix GOOGLE_STORAGE_BUCKET setting

* fix docker-compose tests

* lookup template_id for channel validation

* fix bug preventing creation of multiple steps by id

* rerun mechanism is working

* "loom run restart" and "--force-rerun"

* bundled inputs

* ip_forward true on gcp instances

* fix migration for m2m task to task-attempt

* fix Task.is_responsive method

* added .travis.yml

* add build-loom-packages to .travis.yaml

* remove .travis.yml

* add mptt to server requirements to support old migrations

* add set pipefail in bin/run-tests.sh

* add Jenkinsfile

* update Jenkinsfile

* remove "source" from Jenkinsfile

* fix loomengine_utils.test.test_file_utils.TestLocalFilePattern.testWildcard

* build docker image

* change Jenkinsfile from declarative to script

* typo in Jenkinsfile

* change Jenkinsfile to declarative syntax

* missing quotes in Jenkinsfile

* inspect env

* set noTags False in Jenkinsfile

* fix parentheses in Jenkinsfile

* set version as GIT_COMMIT first 10 chars

* revert Jenkinsfile to default checkout

* set-version handles missing doc

* remove versions from source control

* add git push to Jenkinsfile

* remove VERSION from Dockerfile

* fix version file handling

* enable docker build with no build-args

* Jenkinsfile runs integration test stage on certain branches

* Jenkinsfile fix tag in docker run

* Jenkinsfile fix tag in docker push

* fix regex in Jenkinsfile

* Jenkinsfile fix regex

* fix Jenkinsfile regex

* fix regex in Jenkinsfile

* fix Jenkinsfile regex

* fix regex in Jenkinsfile

* Jenkinsfile add buildingTag to integration test stage

* Jenkinsfile conditional LOOM_VERSION either tag or commit

* Jenkinsfile rewrite LOOM_VERSION logic

* convert Jenkinsfile to scripted

* debug Jenkinsfile

* revert to declarative Jenkinsfile format

* add deployment and integration tests

* fix Jenkinsfile typo

* fix Jenkinsfile typo

* Jenkinsfile installs loom client locally

* rename postBuild-->post in Jenkinsfile

* Jenkinsfile fix path for set-version.sh

* fix typos in Jenkinsfile

* remove dependency on gsutil and gcloud from client

* Jenkinsfile authenticate loom user

* Jenkinsfile fix pw_generator

* Jenkinsfile get Loom admin credentials from jenkins

* Jenkinsfile set username and password on loom server start

* add GOOGLE_APPLICATION_CREDENTIALS to Jenkinsfile environment

* Jenkinsfile typo

* Jenkinsfile increase timeout on integration tests

* use ssh-keys, not sshKeys, to not override GCE project keys

* fix typo in gcloud_start_server playbook

* add refresh_inventory to get instance metadata

* add missing ssh_private_key_path var in playbooks

* revert changes that removed gcloud from ansible

* remove "ansible-generated" note from ssh keys

* Tag Docker image with the current branch

* typo in jenkinsfile

* Jenkinsfile wrap conditional steps in script block

* Jenkinsfile remove redundant tag in docker build

* fix invalid LOOM_SERVER_NAME in Jenkinsfile

* Jenkinsfile configure email

* Jenkinsfile add loom server cleanup

* Jenkinsfile fix typo

* reenable task_attempt cleanup

* Jenkinsfile typos

* Jenkinsfile typo

* Jenkinsfile fix email formatting

* Jenkinsfile add confirm-server-name to loom server delete

* fix task_attempt cleanup, refactor async

* fix unittests

* Jenkinsfile release to PyPi if tag exists

* Jenkinsfile add smoketest to pypi release

* fix Jenkinsfile syntax

* Jenkinsfile replace disallowed chars in LOOM_SERVER_NAME

* Fix Jenkinsfile syntax

* debug Jenkinsfile

* debug Jenkinsfile

* validate version during build

* allow alphanumeric version for dev builds

* typo in Jenkinsfile

* fixed export of file array inputs in worker

* add debug env output to Jenkinsfile

* fix Jenkinsfile syntax

* Jenkinsfile make LOOM_SERVER_NAME all lowercase

* fix retries

* cleanup tmp working directory after TaskAttempt

* delete token on loom server disconnect

* fix retries (#513)

* change WORKING_DIR settings in task_monitor inputs/outputs

* Make "loom template import missing.file" raise an error

* raise ValidationError for duplicate step name

* improve client message for google.cloud.exceptions.InternalServerError

* improve message for "loom server start" --extra-settings with invalid format

* fix import error

* fix import errors

* never retry on google.cloud.exceptions.Forbidden

* check for duplicate user_inputs in run validation

* CSRF_COOKIE_SECURE and SESSION_COOKIE_SECURE

* require LOOM_HTTPS_PORT_ENABLED if LOOM_LOGIN_REQUIRED

* eliminate worker scratch disks

* fix retries for resolving EPEL repo

* handle preexisting TaskAttempts on Run import

* respect --force-rerun on retries

* Task serializer does not require data_path

* added cleanup to RunSerializer

* cascade deletion to run/template children unless protected

* LOOM_DISABLE_DELETE prevents object deletion

* do not mount .ssh dir on server

* add debug info

* increase time for ssh keys to propagate to GCP instance

* 5s timeout on status requests from client

* fix timeout arg in unittests

* Client errors with pretty message, no traceback

* validate tags and labels as alphanumeric

* fix import error

* create bootdisk and instance in one step to avoid leaving disk on failure

* Extend interactive test timeout

* remove "disk_type" setting, not supported by ansible gce module

* create settings bundle and copy it to server

* show server configuration errors in terminal

* Client prompts to login if 403 error

* update unittests

* introduce timeout to kill long-running tasks

* no traceback on client init errors

* include both TaskAttempt containers in cleanup

* refactor client tests

* fix cleanup and error msg for runs with missing inputs

* update unit test

* refactor integration test

* lock down kombu version

* change how template import/export handles steps

* add playbooks to MANIFEST.in for client package

* fix recursive-include syntax

* handle duplicate TaskAttempts on run import

* rename test settings

* migrate prefetch methods to models

* preserve relative file path on export/import

* fix unittests

* refactor serializers inheritance

* fix get_data_nodes on task_attempts

* FilterMixin can lookup multiple references at once

* add filter options back to FileResource

* partial work on RunSerializer

* partial work on RunSerializer - connect i/o

* partial work on Runserializer - import

* partial work on RunSerializer - bulk_create DataNodes

* partial work on RunSerializer - bulk_create DataObjects

* refactor async tasks

* completed RunSerializer refactor

* update unit tests

* remove "sleep" from delayed TaskAttempt retries

* Allow delayed save when creating DataNodes

* InputCalculator works per run, no more target channel

* update requirements

* require requests>=2.5.0 in loom_utils

* add pycrypto to dependencies

* display Loom version and template md5 in browser (#547)

* Added documentation on output parser and glob strings (#548)

* clarify error for duplicate file names in a task (#549)

* "loom server stop" uses internal ip when needed (#550)

* Prerelease 0.7.0 (#551)

* update release notes for 0.7.0

* fix PyPi test, doc version

* update jenkins configuration

* correct typo in Jenkinsfile

* client and worker pass pycodestyle checks (#553)

* Issue 554 (#557)

* remove comma from SQL CASE statements

* set persistent_boot_disk=True, fixes ignored disk size setting

* add elasticsearch and curator to loom server dependencies

* fix sql syntax in case statements

* Prerelease 0.7.1 (#558)

* update releasenotes for 0.7.1, Jenkins runs full integration tests for tagged releases

* correct jenkins stage name for integration tests

* Add doc build test

* allow integration tests to run with self-signed cert

* bugfix matching DataNodes to TaskOutputs for bulk_create

* Task.create_unsaved_task_from_input_set returns correct types if no new task

* fix bug in output routing for parallel steps

* Use full git hash as version bc setuptools normalization edits short hash
Latest commit 1033757 Sep 6, 2018

README.rst

What is loom?

Loom is a platform-independent tool to create, execute, track, and share workflows.

Why use Loom?

Ease of use

Loom runs out-of-the-box locally or in the cloud.

Repeatable analysis

Loom makes sure you can repeat your analysis months and years down the road after you've lost your notebook, your data analyst has found a new job, and your server has had a major OS version upgrade.

Loom uses Docker to reproduce your runtime environment, records file hashes to verify analysis inputs, and keeps fully reproducible records of your work.

Traceable results

Loom remembers anything you ever run and can tell you exactly how each result was produced.

Portability between platforms

Exactly the same workflow can be run on your laptop or on a public cloud service.

Open architecture

Not only is Loom open source and free to use, it uses an inside-out architecture that minimizes lock-in and lets you easily share your work with other people.

  • Write your results to a traditional filesystem or object store and browse them outside of Loom
  • Publish your tools as Docker images
  • Publish your workflows as simple, human-readable documents
  • Collaborate by sharing your workflows and results between Loom servers
  • Connect Loom to multiple file stores without creating redundant copies
  • Efficient re-use of results for redundant analysis steps

How many times do you really need to run the same analysis on the same inputs? Loom knows which steps in your workflow have already been run and seamlessly integrates previous results with the current run, while still maintaining data provenance and traceability.

Graphical user interface

While you may want to automate your analysis from the command line, a graphical user interface is useful for interactively browsing workflows and results.

Security and compliance

Loom is designed with clinical compliance in mind.

Who needs Loom?

Loom is built for the kind of workflows that bioinformaticians run -- multi-step analyses with large data files passed between steps. But nothing about Loom is specific to bioinformatics.

Loom is scalable and supports individual analysts or large institutions.

Get started

Check out our Getting Started Guide and give Loom a try.

http://loom.readthedocs.io/en/latest/installing.html

What is the current status?

Loom is under active development. To get involved, contact info@loomengine.org

Contributors

  • Nathan Hammond
  • Isaac Liao