allow the user to specify the XA drop ratio via the -z option#294
Merged
Merged
Conversation
nh13
commented
Aug 12, 2020
Merged
9 tasks
nh13
added a commit
to fg-labs/bwa-mem3
that referenced
this pull request
Apr 25, 2026
…rder) (#35) * feat(mem): add -z FLOAT option to set XA_drop_ratio Port of bwa PR lh3/bwa#294. The mem_opt_t::XA_drop_ratio field already existed (default 0.80) and was honored in mem_gen_alt via get_pri_idx, but there was no CLI knob to change it. Add -z to the getopt_long string and the help text so users can adjust the minimum score fraction used with -h when populating the XA tag. * feat(mem): add -u flag to emit score,mapq in XA records (XB) Port of bwa PR lh3/bwa#293 (plus the XB base introduced upstream in the same line of work). Adds MEM_F_XB (0x2000) and a -u CLI switch. When set, mem_gen_alt appends ',<score>,<mapq>' to each XA:Z record. The tag name remains XA:Z for downstream compatibility; -u only widens the per-hit field layout from NM to NM,score,mapq. The BAM writer already copies p.XA verbatim into the XA:Z aux tag, so the wider record carries through automatically in --bam mode. * feat(mem): emit MQ:i: mate mapping quality tag Port of bwa PR lh3/bwa#330. Adds MQ:i: to both the SAM text path (mem_aln2sam) and the BAM path (mem_aln_to_bam) for parity. MQ is emitted whenever the mate alignment record is present, matching the upstream semantics. * fix(sam): emit @hd before @sq in bwa_print_sam_hdr Port of bwa PR lh3/bwa#345. Previously @sq was written first and any user-supplied hdr_line (possibly starting with @hd) was appended at the end, producing a SAM stream that violated the spec (the HD line must precede all @sq). Count @hd occurrences in hdr_line, emit the user's @hd up front (falling back to a default "@hd\tVN:1.5\tSO:unsorted\tGO:query" when absent), then the @sq lines, then the remainder of hdr_line only when it did not already carry an @hd — otherwise we would duplicate it. The BAM writer path builds its header via sam_hdr_add_line and already emits @hd first, so it was unaffected. Also drops the dead #if ORIG/#else/#endif branch in this block; the non-ORIG form is the only correct one now. * feat(sam): load SAM headers from <prefix>.hdr or <baseprefix>.dict Port of lh3/bwa PR #348 (jmarshall): at mapping time, look for <prefix>.hdr (or <baseprefix>.dict, stripping a trailing .fa/.fna/.fasta and optional .gz) next to the index and use its contents as SAM header records. This lets users attach rich @sq metadata (AS/M5/SP/AH/...) and @RG/@PG/@co records to a prebuilt index without re-running mem with -H every time. `samtools dict` output is a suitable starting point. Precedence (matches lh3/bwa#348 for the SAM text path): @hd: user's -H > index's .hdr/.dict > "@hd\tVN:1.5\tSO:unsorted\tGO:query". @sq: if -H provides any @sq, use them alone — the index file is ignored entirely; else index's @sq; else generated from bns. Other lines (@RG/@PG/@co): all of -H's and (when not suppressed) the index file's, then bwa_pg. Implementation: - src/bwa.cpp: bwa_print_sam_hdr is refactored into helpers remove_line, has_SQ, count_SQ, print_sam_hdr (all carrying the FILE *fp that bwa-mem2's variant emits to, not stdout via err_printf). A new bwa_load_hdr_from_index(prefix) probes <prefix>.hdr then <baseprefix>.dict and returns a malloc'd, newline-separated but not newline-terminated buffer (caller frees). A new bwa_print_sam_hdr2 entry point takes the already-loaded index header content so the SAM and BAM paths can share a single load. - src/fastmap.cpp: after ref_prefix is finalized (honoring --meth's .bwameth.c2t suffix), load the index header content once via bwa_load_hdr_from_index and route it to both the SAM text path (bwa_print_sam_hdr2) and the --bam path (bam_writer_open). - src/bam_writer.cpp / .h: bam_writer_open gains an idx_hdr_lines parameter. When idx_hdr_lines supplies @sq, the auto-generated @sq from bns is skipped to avoid htslib's duplicate-SN de-dup rejecting the header. When idx_hdr_lines or the user's -H supplies @hd, the default @hd is skipped (htslib's sam_hdr_add_lines does not de-dup @hd). When both the user's -H and the index supply @hd, the index's @hd is filtered out before sam_hdr_add_lines so user -H wins. The caller suppresses idx_hdr_lines entirely when the user's -H supplies @sq, matching the SAM text precedence. This refactor supersedes the narrower @HD-emit-before-@SQ fix in the prior fix(sam) commit on this branch; that commit stays as its own history entry and this one builds cleanly on top of it. Skipped from upstream: bwa.1 manpage and bwase.c/bwape.c hunks, which don't apply to bwa-mem2.
nh13
added a commit
to fg-labs/bwa-mem3
that referenced
this pull request
May 13, 2026
…eflight Closes the gap between the v0.2.0-pre output surface and the docs. - user-guide/output.md: added MQ:i (mate MAPQ, PR #35 via lh3/bwa#330) to the standard tags reference, plus a note that XA:Z widens from chr,pos,CIGAR,NM to chr,pos,CIGAR,NM,score,mapq when -u is passed (PR #35 via lh3/bwa#293). - getting-started/quick-align.md: added MC:Z and MQ:i to the tag table. - performance/overview.md: added 5-rep wall-time median table from the bwa-mem3-bench run at SHA dc7fcfe (2026-05-13) across c6a / c7a / c7i / c7g / c8g for wgs-5M / wes-5M / panel-twist-5M, with the 100.0000%-concordance and NEON-vs-x86 byte-identity statements and the per-arch CV noise envelope. - cli/shm.md: documented the statvfs("/dev/shm") preflight added in PR #86: the new [E::bwa_shm_stage] error format (vs the previous bare [fread] Bad address), the non-fatal failure semantics, and the AWS remount sizing hint. - whats-different/upstream-prs.md: added six previously-missing rows: - @HD-before-@SQ ordering (PR #35 / lh3/bwa#345) under Correctness - mem_matesw shm-ref SIGSEGV (PR #85) under Correctness - -z XA_drop_ratio CLI knob (PR #35 / lh3/bwa#294) under Features - -u XA:Z widening (PR #35 / lh3/bwa#293) under Features - MQ:i tag (PR #35 / lh3/bwa#330) under Features - Bismark XR/XG/XM tag set (PR #90) under Features Bumped the intro's "current as of" stamp from 0.1.0-pre to 0.2.0-pre.
nh13
added a commit
to fg-labs/bwa-mem3
that referenced
this pull request
May 13, 2026
* docs(simd): rewrite for single-binary in-process dispatch (PR #83) Replaces the multi-binary execv launcher description with the single-binary, per-tier-kernel, in-process dispatch design that PR #83 shipped. Sweeps every collateral page that referenced make multi, runsimd.cpp, or bwa-mem3.<tier> companion binaries. - developer-guide/launcher.md: full rewrite as "Single-binary SIMD dispatch (x86)". Covers symbol mangling, per-tier kernel TUs, IBandedPairWiseSW / Ikswv interface split, C-linkage ksw_* wrappers, BWAMEM3_FORCE_TIER / BWAMEM3_DEBUG_SIMD env vars, host-floor precheck integration (PR #95), per-tier parity validation, and a trade-off table vs the prior execv design. - developer-guide/simd-dispatch.md: rewrote intro, dispatch flowchart, and "Adding a new SIMD kernel" checklist. Kept the simd_compat.h abstraction section (still relevant). Added "Per-tier compilation and symbol mangling" section. - performance/simd-dispatch.md: rewrote as concise user-facing page describing the in-process startup flow, BASELINE_ARCH semantics, runtime overrides, and the rationale for in-process vs separate binaries. - SUMMARY.md: renamed "Multi-binary launcher (x86)" entry. - index.md, user-guide/tips.md, user-guide/indexing.md, developer-guide/neon-port.md, developer-guide/building.md, best-practices/build.md, best-practices/anti-patterns.md, performance/tuning.md, performance/pgo.md, reference/glossary.md: removed references to make multi, runsimd.cpp, bwa-mem3.<tier> companion binaries; replaced "use the right binary" advice with bwa-mem3 version / BWAMEM3_DEBUG_SIMD / BWAMEM3_FORCE_TIER. - whats-different/arch-support.md, whats-different/build-infra.md: annotated historical PR entries (#16, #50) with the PR #83 transition context so they accurately describe both the historical and current state. * docs(meth): align YS/YC/YD references with Bismark XR/XG/XM emission (PR #90) PR #90 made YS:Z / YC:Z / YD:Z internal-only carriers (suppressed at BAM emit in bam_writer.cpp under --meth) and emits the Bismark-compatible XR:Z / XG:Z / XM:Z tag set instead. The reference page methylation/tags.md was updated in #90 but five collateral pages still described the old bwameth tag set as user-visible output. - methylation/conversion.md: reframed YS:Z/YC:Z as internal carriers with explicit "suppressed by bam_writer.cpp" note; updated "See also" link; retained the SEQ-restoration mechanism description (which is correctly named at the internal level). - methylation/bwameth-mapping.md: replaced the YS/YC/YD field-compat row with the Bismark tag-set difference; rewrote the "Inline BAM post-processing" paragraph; fixed "See also" link. - best-practices/output-format.md: updated the methylation post-processing paragraph. - whats-different/features.md: reframed the --meth pipeline description to name the emitted XR/XG/XM tags and label YS/YC as internal carriers. Also separately addressed in this branch (whats-different/anti-patterns.md in the prior commit's sweep): the "Mixing bwa-mem3 and bwa-mem2 outputs" warning, which previously claimed bwa-mem3 emits YS/YC/YD. * docs(output): document MQ:i, XA:Z widening, perf numbers, /dev/shm preflight Closes the gap between the v0.2.0-pre output surface and the docs. - user-guide/output.md: added MQ:i (mate MAPQ, PR #35 via lh3/bwa#330) to the standard tags reference, plus a note that XA:Z widens from chr,pos,CIGAR,NM to chr,pos,CIGAR,NM,score,mapq when -u is passed (PR #35 via lh3/bwa#293). - getting-started/quick-align.md: added MC:Z and MQ:i to the tag table. - performance/overview.md: added 5-rep wall-time median table from the bwa-mem3-bench run at SHA dc7fcfe (2026-05-13) across c6a / c7a / c7i / c7g / c8g for wgs-5M / wes-5M / panel-twist-5M, with the 100.0000%-concordance and NEON-vs-x86 byte-identity statements and the per-arch CV noise envelope. - cli/shm.md: documented the statvfs("/dev/shm") preflight added in PR #86: the new [E::bwa_shm_stage] error format (vs the previous bare [fread] Bad address), the non-fatal failure semantics, and the AWS remount sizing hint. - whats-different/upstream-prs.md: added six previously-missing rows: - @HD-before-@SQ ordering (PR #35 / lh3/bwa#345) under Correctness - mem_matesw shm-ref SIGSEGV (PR #85) under Correctness - -z XA_drop_ratio CLI knob (PR #35 / lh3/bwa#294) under Features - -u XA:Z widening (PR #35 / lh3/bwa#293) under Features - MQ:i tag (PR #35 / lh3/bwa#330) under Features - Bismark XR/XG/XM tag set (PR #90) under Features Bumped the intro's "current as of" stamp from 0.1.0-pre to 0.2.0-pre. * docs(release): populate NEWS.md 0.2.0-pre; strengthen release process - NEWS.md: rewrote the 0.2.0-pre entry to cover the full release window. Previous entry only mentioned PR #90; now lists PR #83 (single-binary in-process dispatch), #84 (BASELINE_ARCH=avx2 default), #85 (matesw SIGSEGV fix), #86 (avx512bw autovec cap + /dev/shm preflight), #88 (FMI backwardExt perf), #90 (Bismark XR/XG/XM tags), and #95 (host- floor precheck). Date stamp bumped from 2026-05-09 to 2026-05-13 (the most recent merge in the release window). - cli/version.md: updated the example output to show the SIMD floor: + SIMD runtime: lines added by PR #95, documented the always-exit-0 property, the stderr [W::bwa-mem3] warning on too-old hosts, and the "version | grep ^SIMD" CI-script pattern that works on hosts that cannot run alignment. - developer-guide/release.md: added three new sections. - Semver policy: explicit rules for what bumps MAJOR / MINOR / PATCH given the project's SAM/BAM output, CLI flags, and host-class surface area. - Release-readiness checklist: build (x86_64 + arm64, BASELINE_ARCH smoke), test (make test + all_tiers_parity.sh), bench (bwa-mem3-bench regression gate PASS, methylation carve-out verified), and docs (mdbook clean, NEWS.md and FG-MAIN-TABLE current, upstream-prs.md rows added). - Post-release verification: Read the Docs build check on /v0.X.Y/, clean-clone version-string check, and the "do not move tags, cut a patch instead" policy. Also: whats-different/arch-support.md fg-main reference annotated with the "renamed to main in 0.1.0-pre" context so it reads correctly post-rename. * docs(overview): regenerate FG-MAIN-TABLE from git log master..main The table was 3 entries when the actual fork-carried commit set on main is 87. Regenerated mechanically from `git log --reverse --no-merges master --format='%h%x09%s' master..main`, with the trailing `(#NN)` PR suffix stripped from each topic and linked in a separate PR column. - 87 rows total (was 3). 71 rows have PR links; 13 direct commits (the bwa-mem3 rename sweep, the first three pre-PR commits, etc.) show — in the PR column. - Added a one-line note above the table pointing readers to upstream-prs.md for per-PR upstream-bwa-mem2 disposition (the detailed PR/issue cross-reference is maintained there, so the FG-MAIN-TABLE itself does not need to duplicate upstream-status data per row). - Removed the now-stale `[pr288]` link reference at the bottom of the file — it was referenced only by the dropped first table row. Verified: `make docs` builds clean (single benign local warning about mdbook-mermaid 0.5.0 being called from mdbook 0.5.2 — CI pins both to matched versions so this does not appear in the RTD or CI build). Overview.html renders 89 <tr> rows (the 87-row table plus headers and surrounding markup). Mermaid diagrams on performance/simd-dispatch.html and developer-guide/simd-dispatch.html render as expected.
nh13
added a commit
to fg-labs/bwa-mem3
that referenced
this pull request
May 15, 2026
* docs(simd): rewrite for single-binary in-process dispatch (PR #83) Replaces the multi-binary execv launcher description with the single-binary, per-tier-kernel, in-process dispatch design that PR #83 shipped. Sweeps every collateral page that referenced make multi, runsimd.cpp, or bwa-mem3.<tier> companion binaries. - developer-guide/launcher.md: full rewrite as "Single-binary SIMD dispatch (x86)". Covers symbol mangling, per-tier kernel TUs, IBandedPairWiseSW / Ikswv interface split, C-linkage ksw_* wrappers, BWAMEM3_FORCE_TIER / BWAMEM3_DEBUG_SIMD env vars, host-floor precheck integration (PR #95), per-tier parity validation, and a trade-off table vs the prior execv design. - developer-guide/simd-dispatch.md: rewrote intro, dispatch flowchart, and "Adding a new SIMD kernel" checklist. Kept the simd_compat.h abstraction section (still relevant). Added "Per-tier compilation and symbol mangling" section. - performance/simd-dispatch.md: rewrote as concise user-facing page describing the in-process startup flow, BASELINE_ARCH semantics, runtime overrides, and the rationale for in-process vs separate binaries. - SUMMARY.md: renamed "Multi-binary launcher (x86)" entry. - index.md, user-guide/tips.md, user-guide/indexing.md, developer-guide/neon-port.md, developer-guide/building.md, best-practices/build.md, best-practices/anti-patterns.md, performance/tuning.md, performance/pgo.md, reference/glossary.md: removed references to make multi, runsimd.cpp, bwa-mem3.<tier> companion binaries; replaced "use the right binary" advice with bwa-mem3 version / BWAMEM3_DEBUG_SIMD / BWAMEM3_FORCE_TIER. - whats-different/arch-support.md, whats-different/build-infra.md: annotated historical PR entries (#16, #50) with the PR #83 transition context so they accurately describe both the historical and current state. * docs(meth): align YS/YC/YD references with Bismark XR/XG/XM emission (PR #90) PR #90 made YS:Z / YC:Z / YD:Z internal-only carriers (suppressed at BAM emit in bam_writer.cpp under --meth) and emits the Bismark-compatible XR:Z / XG:Z / XM:Z tag set instead. The reference page methylation/tags.md was updated in #90 but five collateral pages still described the old bwameth tag set as user-visible output. - methylation/conversion.md: reframed YS:Z/YC:Z as internal carriers with explicit "suppressed by bam_writer.cpp" note; updated "See also" link; retained the SEQ-restoration mechanism description (which is correctly named at the internal level). - methylation/bwameth-mapping.md: replaced the YS/YC/YD field-compat row with the Bismark tag-set difference; rewrote the "Inline BAM post-processing" paragraph; fixed "See also" link. - best-practices/output-format.md: updated the methylation post-processing paragraph. - whats-different/features.md: reframed the --meth pipeline description to name the emitted XR/XG/XM tags and label YS/YC as internal carriers. Also separately addressed in this branch (whats-different/anti-patterns.md in the prior commit's sweep): the "Mixing bwa-mem3 and bwa-mem2 outputs" warning, which previously claimed bwa-mem3 emits YS/YC/YD. * docs(output): document MQ:i, XA:Z widening, perf numbers, /dev/shm preflight Closes the gap between the v0.2.0-pre output surface and the docs. - user-guide/output.md: added MQ:i (mate MAPQ, PR #35 via lh3/bwa#330) to the standard tags reference, plus a note that XA:Z widens from chr,pos,CIGAR,NM to chr,pos,CIGAR,NM,score,mapq when -u is passed (PR #35 via lh3/bwa#293). - getting-started/quick-align.md: added MC:Z and MQ:i to the tag table. - performance/overview.md: added 5-rep wall-time median table from the bwa-mem3-bench run at SHA dc7fcfe (2026-05-13) across c6a / c7a / c7i / c7g / c8g for wgs-5M / wes-5M / panel-twist-5M, with the 100.0000%-concordance and NEON-vs-x86 byte-identity statements and the per-arch CV noise envelope. - cli/shm.md: documented the statvfs("/dev/shm") preflight added in PR #86: the new [E::bwa_shm_stage] error format (vs the previous bare [fread] Bad address), the non-fatal failure semantics, and the AWS remount sizing hint. - whats-different/upstream-prs.md: added six previously-missing rows: - @HD-before-@SQ ordering (PR #35 / lh3/bwa#345) under Correctness - mem_matesw shm-ref SIGSEGV (PR #85) under Correctness - -z XA_drop_ratio CLI knob (PR #35 / lh3/bwa#294) under Features - -u XA:Z widening (PR #35 / lh3/bwa#293) under Features - MQ:i tag (PR #35 / lh3/bwa#330) under Features - Bismark XR/XG/XM tag set (PR #90) under Features Bumped the intro's "current as of" stamp from 0.1.0-pre to 0.2.0-pre. * docs(release): populate NEWS.md 0.2.0-pre; strengthen release process - NEWS.md: rewrote the 0.2.0-pre entry to cover the full release window. Previous entry only mentioned PR #90; now lists PR #83 (single-binary in-process dispatch), #84 (BASELINE_ARCH=avx2 default), #85 (matesw SIGSEGV fix), #86 (avx512bw autovec cap + /dev/shm preflight), #88 (FMI backwardExt perf), #90 (Bismark XR/XG/XM tags), and #95 (host- floor precheck). Date stamp bumped from 2026-05-09 to 2026-05-13 (the most recent merge in the release window). - cli/version.md: updated the example output to show the SIMD floor: + SIMD runtime: lines added by PR #95, documented the always-exit-0 property, the stderr [W::bwa-mem3] warning on too-old hosts, and the "version | grep ^SIMD" CI-script pattern that works on hosts that cannot run alignment. - developer-guide/release.md: added three new sections. - Semver policy: explicit rules for what bumps MAJOR / MINOR / PATCH given the project's SAM/BAM output, CLI flags, and host-class surface area. - Release-readiness checklist: build (x86_64 + arm64, BASELINE_ARCH smoke), test (make test + all_tiers_parity.sh), bench (bwa-mem3-bench regression gate PASS, methylation carve-out verified), and docs (mdbook clean, NEWS.md and FG-MAIN-TABLE current, upstream-prs.md rows added). - Post-release verification: Read the Docs build check on /v0.X.Y/, clean-clone version-string check, and the "do not move tags, cut a patch instead" policy. Also: whats-different/arch-support.md fg-main reference annotated with the "renamed to main in 0.1.0-pre" context so it reads correctly post-rename. * docs(overview): regenerate FG-MAIN-TABLE from git log master..main The table was 3 entries when the actual fork-carried commit set on main is 87. Regenerated mechanically from `git log --reverse --no-merges master --format='%h%x09%s' master..main`, with the trailing `(#NN)` PR suffix stripped from each topic and linked in a separate PR column. - 87 rows total (was 3). 71 rows have PR links; 13 direct commits (the bwa-mem3 rename sweep, the first three pre-PR commits, etc.) show — in the PR column. - Added a one-line note above the table pointing readers to upstream-prs.md for per-PR upstream-bwa-mem2 disposition (the detailed PR/issue cross-reference is maintained there, so the FG-MAIN-TABLE itself does not need to duplicate upstream-status data per row). - Removed the now-stale `[pr288]` link reference at the bottom of the file — it was referenced only by the dropped first table row. Verified: `make docs` builds clean (single benign local warning about mdbook-mermaid 0.5.0 being called from mdbook 0.5.2 — CI pins both to matched versions so this does not appear in the RTD or CI build). Overview.html renders 89 <tr> rows (the 87-row table plus headers and surrounding markup). Mermaid diagrams on performance/simd-dispatch.html and developer-guide/simd-dispatch.html render as expected.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.