Samtools to pysam #5037

mvdbeek · 2017-11-19T14:02:53Z

This PR changes all uses of samtools within the datatypes (except for the DataProviders) to pysam, which is a galaxy dependency and therefore doesn't need to be satisfied by conda. This allows us to drop the samtools requirement from the upload and set_metadata tools.

samtools is still required for trackster, which I believe makes use of DataProviders. ~~In principle pysam could also generate .csi indexes for Bcf files, but it seems that this isn't exposed in the pysam wrapper.~~ bcftools is still required because pysam.bcftools.concat outright crashes and exits the python interpreter.

The text was updated successfully, but these errors were encountered:

mvdbeek · 2017-11-19T19:49:34Z

It looks like there are some problems with running pysam.index() within the doctests on linux. The tests pass fine on OSX, but I can reproduce the failure with a docker image. Constructing a simple doctest that runs only pysam.index() works fine, so I am starting to think this is something more complex, perhaps with things that we import. Sometimes I see index: invalid option -- '8' or index: invalid option -- '[' or similar, so perhaps something is going wrong when converting python strings to c strings?

jmchilton · 2017-11-20T01:36:37Z

doctests are terrible anyway for the most part IMO - I'd just convert these to unit tests if you think that would help. Let me know if you'd like me to work on that - I'd be happy to this is important work.

Edit: upon re-reading your comments I'm realizing I probably misinterpreted them as being about doctest specific string handling. Opps. 😅

mvdbeek · 2017-11-20T07:38:17Z

Edit: upon re-reading your comments I'm realizing I probably misinterpreted them as being about doctest specific string handling. Opps

That was one thing I did think about after seeing pysam-developers/pysam#245 (comment), so maybe that was shining through somehow :D.

I have been playing around with the pysam sources, and I can make the failing test pass by changing the option parsing, but I haven't understood at all what I'm doing.
I'll make the tests unit tests, that's better anyway.

mvdbeek · 2017-12-04T06:56:22Z

Alright, with the switch to unit tests the error has gone away!

mvdbeek · 2017-12-06T09:57:50Z

Meh, looks like the gff_to_tabix, bed_to_tabix and interval_to_tabix converters have never worked, since they require 2 input datasets, which will cause the converters to fail with

Specify a dataset of the required format / build.

nsoranzo · 2017-12-06T10:54:20Z

I have this in my git stash that may be helpful for debugging converters: https://gist.github.com/nsoranzo/3618bbc0699ed43aa3e58a065d38e981

dannon · 2017-12-06T12:46:28Z

@mvdbeek These have been used in trackster for a long time, no? After swapping the one I did, it seemed to work in trackster, for me, at least. I think the second dataset is just a converted step, and the first one is kept as an input for for tracking purposes?

mvdbeek · 2017-12-06T13:00:51Z

so it's bed->bgzip -> tabix ? That's kinda wasteful given that creating a tabix ~~file~~ index (seems to) always create a compressed variant of the input. In any case they don't work in the UI, which isn't good either. I'll have to take a closer look.

mvdbeek · 2017-12-06T13:36:50Z

it' actually interval -> bigwig -> tabix, and then fails when accessing locations in the converted files with:

galaxy.webapps.galaxy.api.datasets ERROR 2017-12-06 14:31:56,424 Error in dataset API at listing contents: could not open index for `/Users/mvandenb/src/galaxy/database/files/000/dataset_510.dat`: could not open index for `/Users/mvandenb/src/galaxy/database/files/000/dataset_510.dat`
Traceback (most recent call last):
  File "/Users/mvandenb/src/galaxy/lib/galaxy/webapps/galaxy/api/datasets.py", line 73, in show
    rval = self._data(trans, dataset, **kwd)
  File "/Users/mvandenb/src/galaxy/lib/galaxy/webapps/galaxy/api/datasets.py", line 242, in _data
    ref_seq=region, mean_depth=mean_depth, **kwargs)
  File "/Users/mvandenb/src/galaxy/lib/galaxy/visualization/data_providers/genome.py", line 189, in get_data
    data_file = self.open_data_file()
  File "/Users/mvandenb/src/galaxy/lib/galaxy/visualization/data_providers/genome.py", line 327, in open_data_file
    index=self.converted_dataset.file_name)
  File "pysam/libctabix.pyx", line 343, in pysam.libctabix.TabixFile.__cinit__
  File "pysam/libctabix.pyx", line 391, in pysam.libctabix.TabixFile._open
IOError: could not open index for `/Users/mvandenb/src/galaxy/database/files/000/dataset_510.dat`

mvdbeek · 2017-12-06T13:57:26Z

and the tabix format we are producing isn't acutally a tabix file, it's a tabix index. This is a bit messy :/

mvdbeek · 2017-12-06T14:29:28Z

So it looks like pysam.TabixFile accepts the index parameters but the index extension still needs to end in .tbi:

In [33]: pysam.TabixFile(filename='/Users/mvandenb/src/galaxy/database/files/000/dataset_527.dat', index='/tmp/index')
---------------------------------------------------------------------------
IOError                                   Traceback (most recent call last)
<ipython-input-33-8b1d4a08ebcd> in <module>()
----> 1 pysam.TabixFile(filename='/Users/mvandenb/src/galaxy/database/files/000/dataset_527.dat', index='/tmp/index')

pysam/libctabix.pyx in pysam.libctabix.TabixFile.__cinit__()

pysam/libctabix.pyx in pysam.libctabix.TabixFile._open()

IOError: could not open index for `/Users/mvandenb/src/galaxy/database/files/000/dataset_527.dat`

In [35]: !cp /Users/mvandenb/src/galaxy/database/files/000/dataset_527.dat.tbi /tmp/index.tbi

In [36]: pysam.TabixFile(filename='/Users/mvandenb/src/galaxy/database/files/000/dataset_527.dat', index='/tmp/index.tbi')
Out[36]: <pysam.libctabix.TabixFile at 0x10f6bc290>

mvdbeek · 2017-12-06T15:03:05Z

I think pysam-developers/pysam#586 should fix this issue.

dannon · 2017-12-07T18:24:10Z

@mvdbeek Hrmm. This is going to mean we have to wait for another release :/

mvdbeek · 2017-12-07T18:25:29Z

Or we copy the index to a temporary location with a .TBI extension, that seems to work

dannon · 2017-12-07T18:32:22Z

Yeah, good call, I'd prefer that over having to delay again.

mvdbeek · 2017-12-08T10:27:14Z

Just noticed that the coodinates that trackster requests through the API are completely off, so it's hard for me to verify our changes are working :/.
Try going to a specific location that you know has a feature, this just doesn't work, if you look at the API requests you can see that the coordinates are wrong.

…ts in the object store

Before pysam-developers/pysam#586 is merged and a new release is out we create a symlink to the tbi file, which is required for creating TabixFile instances. Since we want to cleanup the symlinks I turned `get_data_file` into a contextmanager. Along the way I also changed many open()/close() calls to `with` statements.

mvdbeek · 2017-12-08T10:49:09Z

lib/galaxy/visualization/data_providers/genome.py

-        return pysam.Tabixfile(self.dependencies['bgzip'].file_name,
-                               index=self.converted_dataset.file_name)
+        # We create a symlnk to the index file. This is
+        # required until https://github.com/pysam-developers/pysam/pull/586 is merged.


Alright, this is already merged :).

mvdbeek · 2017-12-08T11:29:36Z

~~This does fail on dev (and presumably other galaxy versions) too, fwiw.~~

Mehh, I just didn't migrate properly when I was developing #4690

mvdbeek · 2017-12-08T14:07:00Z

Putting this back in review, I think this is OK now. There are still some things that aren't great, like converters that work only when triggered in trackster, but I think that's for another PR.

This renames some variables to make it clearer what files they reflect. Also adds a very basic test that this works as intended.

jmchilton · 2017-12-08T16:42:38Z

Huge 👍 from me - thanks so much @mvdbeek.

nsoranzo · 2017-12-08T16:57:16Z

lib/galaxy/datatypes/binary.py

    @staticmethod
    def merge(split_files, output_file):
+        """
+        Merges Bam files


s/Bam/BAM/ here and below.

nsoranzo · 2017-12-08T16:57:16Z

lib/galaxy/datatypes/binary.py

+            with open(os.devnull, 'w') as devnull:
+                subprocess.check_call(cmd, stderr=devnull, shell=False)
+            needs_sorting = False
+        except Exception:


s/Exception/subprocess.CalledProcessError/

nsoranzo · 2017-12-08T16:57:16Z

lib/galaxy/datatypes/binary.py

-                raise Exception("Error Grooming BAM file contents: %s" % stderr)
-            else:
-                print(stderr)
+        sorted_file_name = "%s.bam" % tmp_sorted_dataset_file_name_prefix  # samtools accepts a prefix, not a filename, it always adds .bam to the prefix


The comment here is outdated, I think.

nsoranzo · 2017-12-08T17:11:16Z

lib/galaxy/datatypes/tabular.py

@@ -15,6 +15,9 @@
 from cgi import escape
 from json import dumps

+import pysam
+import pysam.bcftools


This import seems unused.

No, that's needed because this isn't imported by default:

In [1]: import pysam In [2]: pysam.bcftools --------------------------------------------------------------------------- AttributeError Traceback (most recent call last) <ipython-input-2-cb47258bfbc9> in <module>() ----> 1 pysam.bcftools AttributeError: 'module' object has no attribute 'bcftools' In [3]: import pysam.bcftools In [4]: pysam.bcftools Out[4]: <module 'pysam.bcftools' from '/Users/mvandenb/.venv/lib/python2.7/site-packages/pysam/bcftools.pyc'>

I mean that pysam.bcftools is used in binary.py , but not in this file.

oh you're right, I initially used this to replace bcftools concat but that didn't work!

No worries, thanks for fixing it! There are 3 small review comments left you may have missed, then it should be ready to merge.

Forgot to add them. Thanks again.

nsoranzo · 2017-12-08T17:24:47Z

lib/galaxy/datatypes/test/1.unsorted.bam seems to be unused now that _is_coordinate_sorted() has been removed.

nsoranzo · 2017-12-08T17:30:06Z

lib/galaxy/datatypes/tabular.py

+    def sniff(self, filename):
+        if not is_gzip(filename):
+            return False
+        return BaseVcf.sniff(self, filename)


BaseVcf.sniff(self, filename) -> super(VcfGz, self).sniff(filename)

nsoranzo · 2017-12-08T17:30:06Z

lib/galaxy/datatypes/tabular.py

+    def sniff(self, filename):
+        if is_gzip(filename):
+            return False
+        return BaseVcf.sniff(self, filename)


BaseVcf.sniff(self, filename) -> super(Vcf, self).sniff(filename)

nsoranzo · 2017-12-08T17:30:06Z

lib/galaxy/visualization/data_providers/genome.py

    def open_data_file(self):
-        return pysam.Tabixfile(self.dependencies['bgzip'].file_name,
-                               index=self.converted_dataset.file_name)
+        # We create a symlnk to the index file. This is


s/symlnk/symlink/

nsoranzo · 2017-12-08T18:25:35Z

test/unit/datatypes/util.py

+
+
+@contextmanager
+def get_dataset(file, index_attr='bam_index', dataset_id=1, has_data=True):


file is a Python reserved word, can you use a different variable name? Also in get_input_files().

nsoranzo · 2017-12-10T13:47:36Z

Fantastic, thanks @mvdbeek!

martenson · 2017-12-11T14:50:05Z

Splendid! Thanks @mvdbeek !!! 💯

mvdbeek added area/datatypes kind/refactoring status/WIP labels Nov 19, 2017

mvdbeek added this to the 18.01 milestone Nov 19, 2017

mvdbeek force-pushed the samtools_to_pysam branch 3 times, most recently from 10ed942 to afbb7f3 Dec 3, 2017

mvdbeek changed the title ~~WIP: Samtools to pysam~~ Samtools to pysam Dec 5, 2017

mvdbeek added status/review and removed status/WIP labels Dec 5, 2017

mvdbeek mentioned this pull request Dec 5, 2017

Tabix indexing fixes. #5131

Closed

mvdbeek added status/WIP and removed status/review labels Dec 6, 2017

Use pysam.merge instead of samtools merge

3fa608f

mvdbeek added 6 commits Dec 8, 2017

one more linting fix

2a6e00d

Need force=True in pysam.tabix_index because the index file path exis…

07158e0

…ts in the object store

We can use multiple threads when converting cram to bam files

64deb4f

Use more threads when sorting BAM files and GALAXY_SLOTS is set

90a56ae

Make sure files don't change in-place

eb636c1

mvdbeek reviewed Dec 8, 2017

View changes

Detect if index symlink hack is necessary

4795e49

mvdbeek force-pushed the samtools_to_pysam branch from c5a1148 to 4795e49 Dec 8, 2017

mvdbeek added status/review and removed status/WIP labels Dec 8, 2017

Decompose interval_to_tabix converter script

46371d1

This renames some variables to make it clearer what files they reflect. Also adds a very basic test that this works as intended.

mvdbeek force-pushed the samtools_to_pysam branch from b53fcfb to 46371d1 Dec 8, 2017

nsoranzo reviewed Dec 8, 2017

View changes

mvdbeek force-pushed the samtools_to_pysam branch from bf1327d to 2805ee0 Dec 10, 2017

Improvements to datatypes suggested by @nsoranzo

309b717

mvdbeek force-pushed the samtools_to_pysam branch from 2805ee0 to 309b717 Dec 10, 2017

nsoranzo merged commit e198dd7 into galaxyproject:dev Dec 10, 2017
6 checks passed

Loading

mvdbeek mentioned this pull request Jan 3, 2018

Add BamNative datatype #5180

Merged

mvdbeek mentioned this pull request Mar 29, 2018

[18.01] Fix metadata setting for CRAM files #5805

Merged

mvdbeek deleted the samtools_to_pysam branch Jun 12, 2018

galaxyproject / galaxy Public

Samtools to pysam #5037

Samtools to pysam #5037

mvdbeek commented Nov 19, 2017 •

edited

Loading

mvdbeek commented Nov 19, 2017 •

edited

Loading

jmchilton commented Nov 20, 2017 •

edited

Loading

mvdbeek commented Nov 20, 2017

mvdbeek commented Dec 4, 2017

mvdbeek commented Dec 6, 2017

nsoranzo commented Dec 6, 2017

dannon commented Dec 6, 2017

mvdbeek commented Dec 6, 2017 •

edited

Loading

mvdbeek commented Dec 6, 2017

mvdbeek commented Dec 6, 2017

mvdbeek commented Dec 6, 2017 •

edited

Loading

mvdbeek commented Dec 6, 2017

dannon commented Dec 7, 2017

mvdbeek commented Dec 7, 2017

dannon commented Dec 7, 2017

mvdbeek commented Dec 8, 2017 •

edited

Loading

mvdbeek Dec 8, 2017 •

edited

Loading

mvdbeek commented Dec 8, 2017 •

edited

Loading

mvdbeek commented Dec 8, 2017

jmchilton commented Dec 8, 2017

nsoranzo Dec 8, 2017

nsoranzo Dec 8, 2017

nsoranzo Dec 8, 2017

nsoranzo Dec 8, 2017

mvdbeek Dec 9, 2017

nsoranzo Dec 10, 2017

mvdbeek Dec 10, 2017

nsoranzo Dec 10, 2017

mvdbeek Dec 10, 2017

nsoranzo commented Dec 8, 2017

nsoranzo Dec 8, 2017

nsoranzo Dec 8, 2017

nsoranzo Dec 8, 2017

nsoranzo Dec 8, 2017

nsoranzo commented Dec 10, 2017

martenson commented Dec 11, 2017



		@contextmanager
		def get_dataset(file, index_attr='bam_index', dataset_id=1, has_data=True):

galaxyproject / galaxy Public

Samtools to pysam #5037

Samtools to pysam #5037

Conversation

mvdbeek commented Nov 19, 2017 • edited Loading

mvdbeek commented Nov 19, 2017 • edited Loading

jmchilton commented Nov 20, 2017 • edited Loading

mvdbeek commented Nov 20, 2017

mvdbeek commented Dec 4, 2017

mvdbeek commented Dec 6, 2017

nsoranzo commented Dec 6, 2017

dannon commented Dec 6, 2017

mvdbeek commented Dec 6, 2017 • edited Loading

mvdbeek commented Dec 6, 2017

mvdbeek commented Dec 6, 2017

mvdbeek commented Dec 6, 2017 • edited Loading

mvdbeek commented Dec 6, 2017

dannon commented Dec 7, 2017

mvdbeek commented Dec 7, 2017

dannon commented Dec 7, 2017

mvdbeek commented Dec 8, 2017 • edited Loading

mvdbeek Dec 8, 2017 • edited Loading

Choose a reason for hiding this comment

mvdbeek commented Dec 8, 2017 • edited Loading

mvdbeek commented Dec 8, 2017

jmchilton commented Dec 8, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nsoranzo commented Dec 8, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nsoranzo commented Dec 10, 2017

martenson commented Dec 11, 2017

mvdbeek commented Nov 19, 2017 •

edited

Loading

mvdbeek commented Nov 19, 2017 •

edited

Loading

jmchilton commented Nov 20, 2017 •

edited

Loading

mvdbeek commented Dec 6, 2017 •

edited

Loading

mvdbeek commented Dec 6, 2017 •

edited

Loading

mvdbeek commented Dec 8, 2017 •

edited

Loading

mvdbeek Dec 8, 2017 •

edited

Loading

mvdbeek commented Dec 8, 2017 •

edited

Loading