BUG: Manifest needs var expansion

[thermokarst]

[nbokulich]

LGTM @thermokarst ! tests run like a charm locally, let's see how busywork feels about it.

[arivers]

I'm the developer for q2-itsxpress. I've been having some users get errors the stem from this file validation module. The issues happen when small, but not empty, fastq files are being written. An example of an error when running qiime in verbose mode is:

Traceback (most recent call last):
  File "/Users/rivers/anaconda3/envs/qiime2-2018.8/lib/python3.5/site-packages/qiime2/plugin/model/file_format.py", line 24, in validate
    self._validate_(level)
  File "/Users/rivers/anaconda3/envs/qiime2-2018.8/lib/python3.5/site-packages/q2_types/per_sample_sequences/_format.py", line 159, in _validate_
    self._check_n_records(record_count_map[level])
  File "/Users/rivers/anaconda3/envs/qiime2-2018.8/lib/python3.5/site-packages/q2_types/per_sample_sequences/_format.py", line 130, in _check_n_records
    % (i * 4 + 1))
qiime2.plugin.model.base.ValidationError: Missing sequence for record beginning on line 17

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/rivers/anaconda3/envs/qiime2-2018.8/lib/python3.5/site-packages/q2cli/commands.py", line 274, in __call__
    results = action(**arguments)
  File "<decorator-gen-450>", line 2, in trim_pair_output_unmerged
  File "/Users/rivers/anaconda3/envs/qiime2-2018.8/lib/python3.5/site-packages/qiime2/sdk/action.py", line 231, in bound_callable
    output_types, provenance)
  File "/Users/rivers/anaconda3/envs/qiime2-2018.8/lib/python3.5/site-packages/qiime2/sdk/action.py", line 390, in _callable_executor_
    spec.qiime_type, output_view, spec.view_type, prov)
  File "/Users/rivers/anaconda3/envs/qiime2-2018.8/lib/python3.5/site-packages/qiime2/sdk/result.py", line 244, in _from_view
    result = transformation(view)
  File "/Users/rivers/anaconda3/envs/qiime2-2018.8/lib/python3.5/site-packages/qiime2/core/transform.py", line 68, in transformation
    self.validate(view)
  File "/Users/rivers/anaconda3/envs/qiime2-2018.8/lib/python3.5/site-packages/qiime2/core/transform.py", line 143, in validate
    view.validate('min')
  File "/Users/rivers/anaconda3/envs/qiime2-2018.8/lib/python3.5/site-packages/qiime2/plugin/model/directory_format.py", line 171, in validate
    getattr(self, field)._validate_members(collected_paths, level)
  File "/Users/rivers/anaconda3/envs/qiime2-2018.8/lib/python3.5/site-packages/qiime2/plugin/model/directory_format.py", line 101, in _validate_members
    self.format(path, mode='r').validate(level)
  File "/Users/rivers/anaconda3/envs/qiime2-2018.8/lib/python3.5/site-packages/qiime2/plugin/model/file_format.py", line 29, in validate
    ) from e
qiime2.plugin.model.base.ValidationError: /var/folders/py/c71jxp_54712mpc0c84zgty40000gn/T/q2-SingleLanePerSamplePairedEndFastqDirFmt-0c21t4va/385_0_L001_R2_001.fastq.gz is not a(n) FastqGzFormat file:

Missing sequence for record beginning on line 17

When I run the sample through the stand-alone version of ITSxpress it produces a forward and reverse fastq with 52 records.

Is this an error that was addressed by the refactoring you just completed? I'm happy to test it locally, but I wasn't sure how to install the latest development version of the whole qiime stack locally.

[thermokarst]

Hey @arivers!

Is this an error that was addressed by the refactoring you just completed?

Nope. The validation message you shared is actually for the FastqGzFormat, not the manifest formats.

I'm happy to test it locally, but I wasn't sure how to install the latest development version of the whole qiime stack locally.

Have you seen this? https://dev.qiime2.org/latest/quickstart/

Missing sequence for record beginning on line 17

This error is when a sequence is written with no nucelotides:

@seq01
AAAA
+
1111
@seq02

+

@seq03
CCCC
+
2222

Is it possible for you to remove the empty records in itsexpress (or the plugin), prior to returning the output?

[arivers]

Fantastic, this will allow me to solve my validation issue! I was thinking the issue was coming from trying to process blank fastq files not blank fastq records. Thanks for pointing me to how to set up the dev environment.

I'll ask these questions over in the qiime developer forum next time, after I posted I remembered that's where most of the developer questions are supposed to go.