Metadata-compatible Manifest formats

[thermokarst]

Improvement Description
The manifest formats are currently a little confusing and cumbersome --- they are a CSV format with three columns: sample-id,absolute-filepath,direction. For paired-end reads this leads to the unfortunate side-effect of having two records per sample, which is a bit confusing. Similarly, for delimited formats in QIIME 2, we tend to lean on TSV over CSV.

This improvement proposes replacing the current manifest format with a Metadata-compatible format, <valid id label>\tabsolute-filepath-forward\tabsolute-filepath-reverse, where the last column (absolute-filepath-reverse) would only be used in the case of paired-end reads.

This has the advantage of being easier to explain (one record per sample), consistent with our Metadata spec, and also allows users to re-use their manifest as metadata later on in their analysis.

Questions

1. Backwards compatibility and deprecation are going to be a hurdle here. Not only are the manifest formats exposed publicly as SingleEndFastqManifestPhred33 et al, but they are also used behind the scenes for the Casava and SingleLanePerSampleSingleEndFastqDirFmt formats. The easiest path forward that I see is to keep the old formats, and keep using the "old" internal manifest format in the existing directory formats. We can define one or more new manifest formats that use the new Metadata-compatible spec, and then it really only changes the import side of things.

cc @gregcaporaso @ebolyen @nbokulich

[ebolyen]

If we can get to transformer transitivity, then backwards compatibility is easy and its just a question of how much technical debt we want to pay off at once. Until then, I like focusing this on just the import side.

[gregcaporaso]

I also think this sounds like a good plan, thanks @thermokarst!