Handle whitespace in TSVTaxonomyFormat -> Metadata transformer

[thermokarst]

The relevant transformer: https://github.com/qiime2/q2-types/blob/master/q2_types/feature_data/_transformer.py#L192-L194

TSVTaxonomyFormat might include leading or trailing whitespace characters. The Metadata constructor doesn't strip whitespace, so this can lead to the following:

...
metadata_column = self._metadata_column_factory(series)
File “/mnt/research/germs/softwares/miniconda2/envs/qiime2-2018.2/lib/python3.5/site-packages/qiime2/metadata/metadata.py”, line 230, in _metadata_c
olumn_factory
column = CategoricalMetadataColumn(series)
File “/mnt/research/germs/softwares/miniconda2/envs/qiime2-2018.2/lib/python3.5/site-packages/qiime2/metadata/metadata.py”, line 632, in init
self._series = self.normalize(series)
File “/mnt/research/germs/softwares/miniconda2/envs/qiime2-2018.2/lib/python3.5/site-packages/qiime2/metadata/metadata.py”, line 758, in normalize
return series.apply(normalize, convert_dtype=False)
File “/mnt/research/germs/softwares/miniconda2/envs/qiime2-2018.2/lib/python3.5/site-packages/pandas/core/series.py”, line 2551, in apply
mapped = lib.map_infer(values, f, convert=convert_dtype)
File “pandas/_libs/src/inference.pyx”, line 1521, in pandas._libs.lib.map_infer
File “/mnt/research/germs/softwares/miniconda2/envs/qiime2-2018.2/lib/python3.5/site-packages/qiime2/metadata/metadata.py”, line 748, in normalize
“whitespace characters: %r” % (cls.name, value))
ValueError: CategoricalMetadataColumn does not support strings with leading or trailing whitespace characters: 'D_0__Bacteria;D_1__Chloroflexi;D_2__Gi
tt-GS-136;D_3__uncultured bacterium ’

This recently came up on the forum: https://forum.qiime2.org/t/qiime-taxa-filter-table-error/3947

[d4straub]

This happened already twice to me. I continued analysis by modifying the taxonomy.qza, which is absolutely not ideal. This bug prohibits reproducible and fluid analysis in my case.

[d4straub]

I am pretty sure that this issue is related to the occurence of # in the SILVA 132 taxonomy files. For example 74 times # in /SILVA_132_QIIME_release/taxonomy/16S_only/99/taxonomy_7_levels.txt and 61 times <whitespace># might lead to around 30 taxas with trailing whitespaces when the taxonomy is truncated.

Is this issue duplicated in #174 ?

[thermokarst]

Hey there @d4straub!

This bug prohibits reproducible and fluid analysis in my case.

Sorry to hear that, PRs are welcome!

I am pretty sure that this issue is related to the occurence of # in the SILVA 132 taxonomy files

Sure, but that is probably a more specific case --- the problem is that the Metadata constructor doesn't allow for any leading or trailing whitespace on values. This applies to any form of Sample of Feature metadata, not just the taxonomy formats. I was on the fence about opening this issue here in this repo, instead of on the framework, but decided to do it here since this taxonomy format issue was a concrete example.

Is this issue duplicated in #174 ?

Not quite --- #174 has to do with the taxonomy IDs, specifically, and how they are consumed in various downstream steps (not necessarily anything to do with qiime2.Metadata --- I think #174 is actually a problem with blast exploding when it has whitespace in IDs).

This issue is referring to the taxonomy values themselves.

Both issues can probably be fixed at the same time by amending a few transformers:

• TSVTaxonomyFormat) -> qiime2.Metadata
• TaxonomyFormat) -> qiime2.Metadata

It might also be possible to fix in the framework, since this is a general problem with the MetadataColumn constructor.

If this is something you are interested in addressing we would love that, and would be happy to provide some guidance, if necessary. Thanks!

[MelissaUribe]

Hello!
I am facing the same problem and all the forums relating to it in the Qiime2 forum are already closed. I am sorry in advance if I am not supposed to ask my specific question regarding this issue here and I hope you can instruct me where it would be appropriate:

So after i got this error I googled and found this possible solution: https://forum.qiime2.org/t/qiime-taxa-filter-table-error/3947/5

I dont fully understand this export function, do you need to create a folder to which the content in the taxonomy.qza is exported to? Anyways, when I try to run it I get this error message:

Error: no such option: --output-dir

I thought if I created a folder called taxonomy-with-spaces it would be solved but I guess I simply dont understand this export function. I tried qiime tools export --help and saw the option presented there is --output-path instead of --output-dir but I just get the same error. Could you help me out please?

Best regards!

[thermokarst]

Hi @DevsLilSis --- can you please post this question to the QIIME 2 Forum? You can open a new post there and provide links to the other (closed) posts you found. Thanks!

[ebolyen]

It does seem like we should probably bump up the priority on this issue. I'll add it to the project backlog for now. It seems simple™.

[thermokarst]

Fixed in #219