SAM Read Group (@RG) is SM required?

[blankenberg]

Looking at the original spec SAM1.pages spec file (e.g: dbdec8b / https://github.com/samtools/hts-specs/raw/dbdec8b0405fa2c4a4708a91722168c7e654bbab/SAM1.pages) both ID and SM are listed as being required (pages formatted file removed mid 2013: 78aa805):

however, when the LaTeX version appeared (07dc1c6) this requirement did not appear.

Certain tools, e.g. Picard Tools, require SM tag in read groups, which matches with the original spec.

Should SM be required or not?

[blankenberg]

@bioinformed good point about versions. The specs are versioned and SAM/BAM files themselves can be 'optionally' versioned with the @HD record (VN is required when @HD is specified).

Looking closer at the pages file, its version is listed asVersion 0.1.2-draft (20090702) whereas the LaTex spec appeared as v1.3 draft.

However, this would be a rather significant specification change that is not mentioned in the version change history appendix, e.g. from version 1.5 (current, https://github.com/samtools/hts-specs/blob/7326e54db05075707fe133d003297681a5313c52/SAMv1.tex ):

\subsection*{1.3: July 2010 to April 2011}

\begin{itemize}
\item Re-add {\tt CC} and {\tt CP} auxiliary tags. (Mar 2011)
\item Add CIGAR N intron/skip operator. (Dec 2010)
\item Add {\tt BQ} BAQ tag. (Nov 2010)
\item Add {\tt RG PG} header field. (Nov 2010)
\item Add BAM description and index sections. (Nov 2010)
\item \textbf{Removal of FLAG letters.} (July 2010)
\end{itemize}

\subsection*{1.0: 2009 to July 2010}

Initial edition.

which indicates that there is an oversight here -- either a failure to label as being required, or a failure to convey the significant change.

[jmarshall]

The transition from a Pages document to LaTeX eight years ago was essentially a complete rewrite by @lh3 (of mostly his own previous document) and the differences were not individually called out in source control — which was barely in use for this spec at the time.

However Heng noted on the mailing list at the time that he did indeed intend this particular relaxation:

RG SM (sample) is no longer a required field like it was in the previous version of the spec, is this intentional?

This is intentional. Sometimes, we simply want to group reads without sample information.

I can see his point. Certainly in the typical usage, clinical or otherwise, a read group corresponds to a library or a sample and sensible pipelines will use the defined @RG LB and SM tags to track that information. However in other circumstances there may not be significant sample information available, and the specification shouldn't unnecessarily preclude the use of readgroups in such circumstances.

Picard in strict mode requires @RG SM (but not e.g. @RG LB), and this code would have been written thus because SM was required in the original spec, as you noted. Picard validates a number of things that are recommended practice for the typical SAM/BAM usage, but are not strictly required by the spec; and this could be considered one of them.

[jkbonfield]

Picard has its own set of definitions about what the Picard tool chain requires, but please do not confuse this with the specification. It has a number of additional restrictions (which may be relaxed via VALIDATION_STRINGENCY) of which this is one.

The SAM spec is in a much better, more tracked, situation since it converted from Pages to LaTeX as it is now version controlled. That in turn means that in a clinical setting a very precise version may be specified and adhered to. Failure to do this isn't a problem caused through fluidity of the spec, but is poor experimental design or failure to follow procedures in a clinical environment. @bioinformed: Face facts - things change and technology marches on. This is what the versioning is for!

[jkbonfield]

@blankenberg I wrote the version history part of the spec, but I did so purely based on git commit history. This means we cannot write anything prior to 1.3 as the spec wasn't in version control then and it'd be too much work to try and figure out all the older changes.

[blankenberg]

Thanks again for the history and clarifications.