Repository navigation
Giraffe server protocol
vg giraffe-server loads the Giraffe indexes once, then reads requests from
standard input and writes answers to standard output until its input ends.
This page describes what a client sends and what it gets back. It is meant to
be enough to write a client without reading vg.
Everything is text. Fields are separated by one tab. Every line ends with one
newline character (\n, not \r\n).
A client sends requests. There are two kinds:
- a read, on one line, to be mapped to the graph; This is the main usage of
vg giraffe-server. - a
SURJECT_WITH_ANCHORSblock, on several lines, to place a graph alignment the client already has on one path of the graph.
The server answers every request with exactly one frame: a result frame or an error frame. A frame carries the name of its request, so a client matches answers to requests by name.
client server
------ ------
r1<TAB>ACGT...<TAB>IIII...
r2<TAB>ACGT...<TAB>IIII...
PROCESS_BATCH
READ<TAB>r1<TAB>1
r1<TAB>150<TAB>0<TAB>150<TAB>+<TAB>>12>13 ...
READ<TAB>r2<TAB>1
r2<TAB>150<TAB>0<TAB>150<TAB>+<TAB>>40>41 ...
Standard error carries log lines for people. They are not part of the protocol, and a client must not parse them.
A read is one line with one to four fields:
SEQUENCE
NAME<TAB>SEQUENCE
NAME<TAB>SEQUENCE<TAB>QUALITY
NAME<TAB>SEQUENCE<TAB>QUALITY<TAB>TARGET
| Field | Rules |
|---|---|
NAME |
Any text without a tab, and not empty. It comes back in the frame. Use a name that no other request waiting for an answer has, or the answers cannot be told apart. A read sent as a bare SEQUENCE is named read_0, read_1, and so on, in the order such reads arrive. |
SEQUENCE |
One or more of A, C, G, T, N (can use lowercases). |
QUALITY |
Base qualities as in FASTQ: one character per base, phred+33, from ! to ~. It is either empty or as long as SEQUENCE. Without qualities, the mapping quality is not capped by base quality, as for FASTA input to vg giraffe. |
TARGET |
The name of a path to surject each alignment of this read onto, exactly as the graph names it. The server must have indexed the path's positions when it started. It indexes the paths given with --surject-target and, when at least one is given, also the reference paths and generic paths of the graph. An empty TARGET is the same as none. With a TARGET, QUALITY may be the empty string between two tabs. |
The server collects reads into a batch and maps the batch at once. A read gets its answer when its batch is mapped, which happens when:
- the batch reaches
--batch-sizereads (256 by default); - the client sends a line with only
PROCESS_BATCH; - a
SURJECT_WITH_ANCHORSblock arrives; - the input ends (client closes the standard input).
A client that waits for answers must therefore send PROCESS_BATCH after its
last read. PROCESS_BATCH itself gets no frame. Empty lines are skipped.
A read with a bad SEQUENCE or QUALITY is not put in the batch. The server
writes its error frame as soon as it reads the line.
This request places a graph alignment on a path whose positions the server has not indexed. The client supplies anchors: places on the target path whose offsets it already knows.
SURJECT_WITH_ANCHORS<TAB>NAME<TAB>TARGET<TAB>PATH_LENGTH<TAB>ANCHOR_COUNT
ANCHOR_COUNT anchor lines
one GAF line
| Header field | Meaning |
|---|---|
NAME |
The name for the frame, as for a read. |
TARGET |
The target path's name. If the graph has no path with this name, the server uses the path that the first anchor lies on. This lets a client name a haplotype loosely, as in HG002#2. |
PATH_LENGTH |
The length of the target path in bases. With 0, the server walks the whole path to measure it, which is slow for a long path. Any other number must be right: the server does not check it. |
ANCHOR_COUNT |
The number of anchor lines that follow. With 0 the server knows no position on the path, and the status is failed. |
An anchor is a run of consecutive steps of the target path. An anchor line has six numbers:
| Column | Meaning |
|---|---|
| 1 | GBWT node of the run's first step. A GBWT node is 2 * node ID, plus 1 for the reverse orientation. |
| 2 | Offset of that step among the visits in the node's GBWT record. Columns 1 and 2 together are a step handle of a GBWTGraph. |
| 3, 4 | The same two numbers for the run's last step. The last step may be the first step. |
| 5 | Offset on the target path of the first base of the first step, counted from 0. |
| 6 | Offset on the target path of the first base of the last step, counted from 0. |
The anchors must be right for the graph the server has loaded. The server
checks that a walk along the path from each anchor's first step reaches its
last step at the offset in column 6, and that the first anchor lies on
TARGET when TARGET is a path name. It answers with an error frame if not.
Together the anchors' runs must cover every step of the target path that the
surjected alignment touches.
The GAF line is one alignment as this server writes it in a result frame. Its
path must be a list of oriented node IDs (>12<13), and it needs the cs tag
that the server writes. Tags that the server added may stay on the line.
The server reads all the lines of a block before it answers, also when the
block has an error, so the next request starts on a line of its own. The one
exception is a header whose ANCHOR_COUNT is not a number, or which does not
have five fields: the server cannot tell how long the block is, so each of the
block's other lines is then read as a request of its own, and gets an error
frame.
Before the first frame, with --emit-header, the server writes the graph's
GAF header lines, if the graph has any. Everything after that is frames.
READ<TAB>NAME<TAB>N
N GAF lines
N is followed by a newline, and then come exactly N lines.
- For a read, there is one line per reported alignment, best first.
Nis at least 1: a read that does not map gets one line with*for its path. - For a
SURJECT_WITH_ANCHORSblock,Nis 1. The line is the GAF line that was sent, with the surjection tags added.
ERROR<TAB>NAME<TAB>MESSAGE
An error frame is one line. MESSAGE is text for a person and has no tab in
it. NAME is * for a line that could not be read as a request, and then
MESSAGE starts with line K:, where K counts input lines from 1.
- The result frames of reads come out in the order the reads were sent.
- The error frame of a read with a bad
SEQUENCEorQUALITY, and of a line that is not a request, is written when the line is read. It can come before the frames of earlier reads that still wait in the batch. - The frame of a
SURJECT_WITH_ANCHORSblock can come at any time after the block was sent: before or after the frames of later requests, and also between two frames of one batch of reads. - A frame is never split. Its lines are not mixed with another frame's.
- A request is one read line or one
SURJECT_WITH_ANCHORSblock. - Every request gets exactly one frame: a result frame or an error frame.
- An error in a request affects only that request. The server goes on.
- A line that is neither a request nor
PROCESS_BATCHgets an error frame with the name*. Empty lines are skipped. - A frame is written whole.
- The server stops when its input ends, after it has answered everything it read, and exits with status 0. Before the first request it also stops, with status 1 and no frame, if its options or indexes are bad. If the server dies, the client sees standard output close, and should treat every request without an answer as failed.
The server gives no sign that it has finished loading. A client that needs to
know sends one read and PROCESS_BATCH, and waits for the frame.
Some of the messages:
| Message | Request |
|---|---|
empty SEQUENCE |
read |
SEQUENCE has characters other than A, C, G, T and N |
read |
QUALITY has character code 32 at offset 4, outside phred+33 ('!' to '~') |
read |
QUALITY length (4) does not match SEQUENCE length (10) |
read |
line 32: expected 1-4 tab-separated fields or a known command, got 5 |
none (*) |
line 33: not a known command or a nucleotide sequence |
none (*) |
line 34: empty NAME |
none (*) |
malformed SURJECT_WITH_ANCHORS header (need 5 tab fields, got 4) |
block |
malformed anchor line 606 (need 6 tab-separated numbers) |
block |
GAF path has node 999, which is not in the graph |
block |
the anchors are not on the target path CHM13#0#chrM |
block |
A read that cannot be mapped for another reason, and a surjection that fails inside vg, also get an error frame, with vg's own message.
Each line is a GAF record as vg giraffe writes it: the twelve GAF columns,
then vg's own tags (AS, bq, cs, dv and others). The server adds the
tags below after those. Their names are local to this server.
A tag with type i holds an integer, and Z holds text.
These say which haplotypes of the graph follow the alignment's path. Every line of a result frame for a read has them.
The server cuts the path into segments. A segment is a run of the path's
nodes that at least one haplotype follows without a break, and it is as long
as it can be. A path that some haplotype follows from end to end is one
segment. A node that no haplotype visits is in no segment. The tags hp to
ht describe the longest segment, the one whose nodes have the most
bases, and the first such segment if there are several.
A haplotype is named sample#phase#contig, as in HG002#1#chr1. This is
shorter than a path name in the graph, which can also have a fragment number:
all the fragments of a haplotype share one name here. A generic path, which
has no sample, is named by its contig alone, as the graph names it. Where the
GBWT has no name for a sample or a contig, its number stands in.
| Tag | Meaning |
|---|---|
hp:Z |
The distinct names of the haplotypes that follow the longest segment, sorted, with commas between them. Empty if there is no segment. |
hn:i |
The number of names in hp. |
hb:Z |
One haplotype from hp that a client can use when it needs a single one, for example as a surjection target. It is a haplotype of a reference sample, or a generic path, if hp has one, and otherwise the one with the lowest sequence ID in the GBWT, so it is the same on every run. * if there is none. |
hf:i |
1 if hb is from a reference sample or is a generic path, else 0. |
hc:i |
The number of times GBWT sequences follow the longest segment. A haplotype that follows it twice counts twice, so hc can be more than hn. |
ht:i |
1 if hp may be incomplete. The server looks up at most 1000 of the hc sequences. |
hl:i |
The node ID of the boundary node before the path, in the path's direction, of the smallest snarl that contains the whole path. |
hr:i |
The same for the boundary node after the path. |
hq:i |
1 if one segment is the whole path, else 0. |
hv:i |
The share of the path that the longest segment covers, in percent, rounded down. Both lengths are sums of whole node lengths. |
hs:i |
The number of segments. |
hm:Z |
Only when hs is more than 1. One entry per segment, in path order, with ; between entries. An entry is LENGTH:COUNT:NAME: the bases in the segment's nodes, the number of haplotype names of the segment, and the segment's own hb. |
hl and hr are each written only when the node was found. To find them the
server takes the haplotypes that follow the whole path and walks out from each
end while they all go to the same node, until the first node outside the
snarl. A side is missing when no haplotype follows the whole path, when no
snarl contains the whole path, when the haplotypes part before they leave the
snarl, or when the walk takes more than 1000 steps.
An unmapped read has hp:Z: (empty), hb:Z:*, and 0 in the other tags.
These say where the alignment lies on one path. A line has them when its read
named a TARGET, and the line of a SURJECT_WITH_ANCHORS answer has them.
sj:Z is always there and holds the status. The other tags are there only
when the status is ok.
sj:Z |
Meaning |
|---|---|
ok |
The alignment was placed on the target path. |
empty |
The alignment has no path in the graph: the read did not map. |
unknown_path |
For a read: the graph has no path with the target's name. For a block: the name is not a path, and there is no anchor to take the path from. |
not_indexed |
The graph has the path, but the server did not index its positions. A server started without --surject-target indexes no path, and gives this status to every line of a read with a TARGET, also when the read did not map or the name is not a path. Only for reads. |
incompatible |
The target path does not visit, in order, the nodes of the alignment's longest segment, or the alignment has no segment. Only for reads. |
failed |
vg found no alignment on the target path. |
| Tag | Meaning |
|---|---|
sn:Z |
The name of the path, as the graph has it. |
sp:i |
Where the alignment starts on the path: the offset, counted from 0, of its leftmost base on the path's forward strand. |
sr:i |
1 if the read is on the path's reverse strand, else 0. |
ss:i |
The alignment score on the path. |
sm:i |
The mapping quality. It is the graph alignment's. |
sc:Z |
The CIGAR string against the path's forward strand, as in SAM. It uses M, I, D and S. A match and a mismatch are both M. * if there is none. |
sM:i |
The number of read bases that match the path. |
sX:i |
The number of read bases that differ from the path. |
sI:i |
The number of read bases that are not on the path. |
sD:i |
The number of path bases that are not in the read. |
The answer to a SURJECT_WITH_ANCHORS block has two more tags when the status
is ok:
| Tag | Meaning |
|---|---|
an:i |
The number of anchors that were sent. |
ap:Z |
The name of the path the server used. It is the same as sn. It matters when TARGET was not a path name. |
A surjection belongs to one graph alignment, and the tags keep the two in one record, so a frame still holds one line per alignment. The cost is that a line can report only one placement. When vg places an alignment on the path in several pieces, the server reports the piece with the best score.
Two reads, the second one with a target, and a line that is not a request. The
server was started with --surject-target CHM13#0#chrM.
r1<TAB>ACGTACGTAC<TAB>IIIIIIIIII
r2<TAB>ACGTACGTAC<TAB><TAB>CHM13#0#chrM
HELLO
PROCESS_BATCH
ERROR<TAB>*<TAB>line 3: not a known command or a nucleotide sequence
READ<TAB>r1<TAB>1
r1<TAB>10<TAB>0<TAB>10<TAB>+<TAB>>7>8<TAB>...<TAB>hp:Z:CHM13#0#chrM,GRCh38#0#chrM<TAB>hn:i:2<TAB>hb:Z:CHM13#0#chrM<TAB>hf:i:1<TAB>hc:i:2<TAB>ht:i:0<TAB>hq:i:1<TAB>hv:i:100<TAB>hs:i:1
READ<TAB>r2<TAB>1
r2<TAB>10<TAB>0<TAB>10<TAB>+<TAB>>7>8<TAB>...<TAB>hs:i:1<TAB>sj:Z:ok<TAB>sn:Z:CHM13#0#chrM<TAB>sp:i:3577<TAB>sr:i:0<TAB>ss:i:20<TAB>sm:i:60<TAB>sc:Z:10M<TAB>sM:i:10<TAB>sX:i:0<TAB>sI:i:0<TAB>sD:i:0
The error frame comes first here because the server answers a bad line as soon
as it reads it, and the two reads only when PROCESS_BATCH arrives. The GAF
lines are shortened: ... stands for the columns and tags that are left out.