The \eslmod{sqio} module handles input from unaligned sequence data files, such as FASTA files. \eslmod{sqio} can automatically recognize and parse several different file formats, including FASTA, UniProt, Genbank, DDBJ, and EMBL. Additionally, it can read individual unaligned sequences from multiple alignment files in several different formats, including Stockholm, Clustal, aligned FASTA, PSI-BLAST, and Phylip. Sequences can be read from normal files, directly from the \ccode{stdin} pipe, or from \ccode{gzip}-compressed files. Sequence files can be automatically looked for in a list of one or more database directories, specified by an environment variable (such as \ccode{HMMERDB}). Table~\ref{tbl:sqio_api} lists the functions in the \eslmod{sqio} API. The module uses an \ccode{ESL\_SQFILE} object which works much like an ANSI C \ccode{FILE}, maintaining information for an open sequence file while it's being read. % API table is auto generated by the Makefile, % using autodoc -t esl_sqio.c % \input{apitables/esl_sqio_api} \subsection{Example: reading sequences from a file} Figure~\ref{fig:sqio_example_text} shows a program that opens a file, reads sequences from it one at a time, then closes the file. \begin{figure} \input{cexcerpts/sqio_example_text} \caption{Example of reading sequences from a file.} \label{fig:sqio_example_text} \end{figure} A FASTA file named \ccode{seqfile} is opened for reading by calling \ccode{esl\_sqfile\_Open(filename, format, env, \&sqfp)}, which creates a new \ccode{ESL\_SQFILE} and returns it through the \ccode{sqfp} pointer. If the \ccode{format} is passed as \ccode{eslSQFILE\_UNKNOWN}, then the format of the file is autodetected. Here, we bypass autodetection by asserting that the file is in FASTA format by passing a \ccode{eslSQFILE\_FASTA} code. (See below for a list of valid codes and formats.) The optional \ccode{env} argument is described below too; here, we're passing \ccode{NULL} and not using it. Several things can go wrong in trying to open a sequence file that are beyond the control of Easel or your application, so it's important that you check the return code. \ccode{esl\_sqfile\_Open()} returns \ccode{eslENOTFOUND} if the file can't be opened, and \ccode{eslEFORMAT} if the file is empty, or if autodetection can't determine its format.\footnote{Additionally, internal errors might be thrown, which you should check for if you installed a nonfatal error handler.} The file is then read one sequence at a time by calling \ccode{esl\_sq\_Read(sqfp, sq)}. This function returns \ccode{eslOK} if it read a new sequence, and leaves that sequence in the \ccode{sq} object that the caller provided. When there is no more data in the file, \ccode{esl\_sq\_Read()} returns \ccode{eslEOF}. If at any point the file does not appear to be in the proper format, \ccode{esl\_sq\_Read()} returns \ccode{eslEFORMAT}. The application must check for this. The API can provide a little information about what went wrong and where. \ccode{sqfp->filename} is the name of the file that we were parsing (not necessarily the same as \ccode{seqfile}; \ccode{sqfp->filename} can be a full pathname if we used an \ccode{env} argument to look for \ccode{seqfile} in installed database directories). The function \ccode{esl\_sqfile\_GetErrorBuf()} should be called to get a pointer to the generated error message. The buffer is a brief explanatory message that gets filled in when a \ccode{eslEFORMAT} error occurs. \footnote{Unlike in the MSA module, you don't get access to the current line text; some of sqio's parsers use fast block-based (\ccode{fread()}) input instead of line-based input.} We can reuse the same \ccode{ESL\_SQ} object for subsequent sequences by calling \ccode{esl\_sq\_Reuse()} on it when we're done with the previous sequence. If we wanted to load a set of sequences, we'd \ccode{\_Create()} an array of \ccode{ESL\_SQ} objects. Finally, to clean up properly, a \ccode{ESL\_SQ} that was created is destroyed with \ccode{esl\_sq\_Destroy(sq)}, and a \ccode{ESL\_SQFILE} is closed with \ccode{esl\_sqfile\_Close()}. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \subsection{Digital sequence input mode} Most Easel-based programs manipulate sequences in Easel's digital sequence format, using \eslmod{alphabet}, as opposed to manipulating them as plaintext. The \eslmod{sqio} reader can be used either in text mode or digital mode. In text mode, you get the \ccode{sq->seq} field of the \ccode{ESL\_SQ}; in digital mode, you get \ccode{sq->dsq}. To use digital mode, both the \ccode{ESL\_SQFILE} reader and the \ccode{ESL\_SQ} sequence object must be set to digital mode. The reader, because it has an input map that maps plaintext input characters to internal residue codes (plaintext or digital), or errors. The sequence object, because it needs to have either its \ccode{seq} or \ccode{dsq} field allocated. Both also carry a copy of the pointer to the alphabet. Figure~\ref{fig:sqio_example_digital} shows an example of the standard idiom for opening files in digital mode, autoguessing their format and alphabet by default, and allowing format and alphabet to be specified on the command line. \begin{figure} \input{cexcerpts/sqio_example_digital} \caption{Standard idiom for reading sequences in digital mode.} \label{fig:sqio_example_digital} \end{figure} %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \subsection{Accepted formats} Accepted unaligned sequence file formats (and their Easel format codes) include: \begin{tabular}{lll} \textbf{name} & \textbf{code} & \textbf{description} \\ fasta & \ccode{eslSQFILE\_FASTA} & FASTA format \\ embl & \ccode{eslSQFILE\_EMBL} & EMBL DNA database format \\ genbank & \ccode{eslSQFILE\_GENBANK} & GenBank DNA database format \\ ddbj & \ccode{eslSQFILE\_DDBJ} & DDBJ DNA database format \\ uniprot & \ccode{eslSQFILE\_UNIPROT} & UniProt protein database format \\ \end{tabular} The above names, case-insensitive, are what a user uses to specify a format on a command line: i.e.\ \ccode{--informat fasta} or \ccode{--informat FASTA}. The codes are what you use as a developer to specify a format to an Easel function call. Additionally, there is a code \ccode{eslSQFILE\_UNKNOWN}. It tells \ccode{esl\_sqfile\_Open()} to perform format autodetection.\footnote{There are some other formats as well, which we don't advertise because they're less well supported.} %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \subsection{Reading from stdin and compressed files} There are two special cases for input files. The module can read sequence input from a stdin pipe. If the \ccode{seqfile} argument is -'', \ccode{esl\_sqfile\_Open()} opens'' standard input (really, it just associates \ccode{stdin}, which is always open, with the \ccode{ESL\_SQFILE}). The module can read compressed sequence files. If the \ccode{seqfile} argument to \ccode{esl\_sqfile\_Open()} ends in \ccode{.gz}, the file is assumed to be compressed with \ccode{gzip}; instead of opening it normally, \ccode{esl\_sqfile\_Open()} opens it as a pipe from \ccode{gzip -dc}. Your system must support pipes to use this.\footnote{Specifically, it must support the \ccode{popen()} system call (POSIX.2 compliant operating systems do). The \ccode{configure} script automatically checks this at compile-time and defines \ccode{HAVE\_POPEN} appropriately.} Obviously, the user must also have \ccode{gzip} installed and in his PATH. For both special cases, the catch is that you can't use format autodetection; you must provide a valid known format code when you read from stdin or from a compressed file. Pipes are not rewindable, and format autodetection currently relies on a two-pass algorithm: it reads partway into the file to determine the format, then rewinds to start parsing for real.\footnote{The \eslmod{msafile} module is more advanced. Its parsers are based on the newer \eslmod{buffer} module which provides rewindable input buffers even for stdin and pipes.} %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %% SRE: commented NCBI section out. %% I don't believe that it works, or that it's up to date. %% Needs testing. %% \subsection{NCBI BLAST database support} %% If sqio is augmented with the ncbi module, then the sqio API gains the %% ability to read NCBI BLAST database formats in addition to ASCII file %% formats. The sqio API remains exactly the same (the caller doesn't %% have to use any msa module functions). %% To open a BLAST database, the format must be supplied. If a format %% of \ccode{eslSQFILE\_UNKNOWN} is specified, only ASCII based format, %% i.e. FASTA, GENBANK, etc. will be opened. %% When opening a BLAST database, the file name should not include any %% extensions. If the \ccode{ESL\_ALPHABET} is not specified, Easel %% will first try to open a protein database, followed by a DNA database %% and finally a multi-volume database. The handling of a multi-volume %% database is done through the alias file. The only directive in %% the alias file is the DBLIST line, listing all the database volumes. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \subsection{Adding a sequence parser} New parsers for new formats can be plugged into \eslmod{sqio} without any API changes. Existing Easel-based programs don't need code changes to use the new parser. A new parser will need a format type, a structure for parser specific data, API functions and a hook into the \ccode{sqfile\_open} function. The list of formats are defined in \ccode{esl\_sqio.h}. A new \ccode{\#define} will be added to the existing formats: \input{cexcerpts/sq_sqio_format} A data structure for parser specific data will need to be added to \ccode{ESL\_SQDATA}. This structure is a union of the different parser specific data structures. \input{cexcerpts/sq_sqio_data} Finally, a set of parser specific function pointers need to be defined. The functions in \ccode{esl\_sqio.c} in turn call these function pointers. The \ccode{esl\_sqfile\_Open} function initializes the function pointers to NULL, so if they are not set, an exception will occur when the function is called. At a minimum, the function should be defined to return an \ccode{eslEUNIMPLEMENTED}. Below is a map the the function pointers to their respective function. \begin{tabular}{ll} \\ Function pointer & \eslmod{sqio} function \\ \hline position & esl\_sqfile\_Position \\ close & esl\_sqfile\_Close \\ set\_digital & esl\_sqfile\_SetDigital \\ guess\_alphabet & esl\_sqfile\_GuessAlphabet \\ read & esl\_sqio\_Read \\ read\_info & esl\_sqio\_ReadInfo \\ read\_seq & esl\_sqio\_ReadSequence \\ read\_window & esl\_sqio\_ReadWindow \\ echo & esl\_sqio\_Echo \\ read\_block & esl\_sqio\_ReadBlock \\ open\_ssi & esl\_sqfile\_OpenSSI \\ pos\_by\_key & esl\_sqfile\_PositionByKey \\ pos\_by\_number & esl\_sqfile\_PositionByNumber \\ fetch & esl\_sqio\_Fetch \\ fetch\_info & esl\_sqio\_FetchInfo \\ fetch\_subseq & esl\_sqio\_FetchSubseq \\ is\_rewindable & esl\_sqfile\_IsRewindable \\ get\_error & esl\_sqfile\_GetErrorBuf \\ \end{tabular} \bigskip A hook needs to be added to the function \ccode{sqfile\_open}. This hook will try to open the specified file. If successful, the \ccode{ESL\_SQFILE} structure should be filled in with function pointers and the parser specific data and the open hook return \ccode{eslOK}. If the sequence files were not found for the specific parser, an \ccode{eslENOTFOUND} is returned and the next parser tries to open the file. Below is an example of code that tries to open an NCBI BLAST database if not successful, then the ASCII sequence parsers try to open the file. \begin{cchunk} if (format == eslSQFILE\_NCBI && status == eslENOTFOUND) status = esl\_sqncbi\_Open(sqfp->filename, sqfp->format, sqfp); if (status == eslENOTFOUND) status = esl\_sqascii\_Open(sqfp->filename, sqfp->format, sqfp); \end{cchunk}