new feature --append https://github.com/gerontec/blastDiamond/blob/main/Diamond_append.md

[gerontec]

With that it is not needed to makedb from scratch. https://github.com/gerontec/blastDiamond/blob/main/Diamond_append.md

On a 2.5 million sequence subset of nr, built two ways:

one makedb run over all 2.5 M sequences
makedb over the first 2.0 M, then makedb --append with the remaining 0.5 M

The resulting .dmnd files are byte-identical, including the database hash.

[gerontec]

How the daily update --append works:

#!/usr/bin/env python3
"""Daily delta for nr_full.dmnd and blast_* from
GenBank daily-nc (ncMMDD.gnp.gz, GenPept, ~180 days retained) and
RefSeq daily (rsnc.MMDD.YYYY.gpff.gz, GenPept, ~21 days retained).

A protein is considered new if its sequence (blake2b-64 of uppercase letters) is neither in nr nor in
a previous delta - NCBI does not list identical proteins separately in nr, matching by accession
would append massive amounts of duplicates.
All open daily files since the last nr release are processed in ONE append run.
"""
import os, re, sys, json, gzip, time, fcntl, shutil, hashlib, subprocess, urllib.request, smtplib
from email.mime.text import MIMEText
import numpy as np

import blast_db, pymysql

E = os.environ.get
PREFIX = E("BLAST_PREFIX", "blast_")
DMND = E("DMND", "/home/gh/diamond/nr_full.dmnd")
DIAMOND = E("DIAMOND_APPEND_BIN", os.path.expanduser("/src/diamond/build/diamond"))
TAXDIR = E("TAXDIR", "/mnt/archive/taxdump")
WORK = E("WORK", "/home/gh/nr_daily")
SEQ_CACHE = E("SEQ_CACHE", "/mnt/archive/nr_seq_hashes.npy")
ACC_CACHE = E("ACC_CACHE", "/mnt/archive/nr_acc_hashes.npy")
LOG = E("DELTA_LOG", os.path.expanduser("/python/nr_daily_delta.log"))
LOCK = E("NR_LOCK", "/home/gh/.nr_dmnd.lock")
TEST_FILES = E("TEST_FILES") # "source|path|YYYY-MM-DD,..." instead of NCBI listing
NO_MAIL = E("NO_MAIL") == "1"
SOURCES = {
"genbank-daily": ("https://ftp.ncbi.nlm.nih.gov/genbank/daily-nc/", r"nc\d{4}.gnp.gz"),
"refseq-daily": ("https://ftp.ncbi.nlm.nih.gov/refseq/daily/", r"rsnc.\d{4}.\d{4}.gpff.gz"),
}

[bbuchfink]

Very well thanks for the effort. If you want me to merge this and get tagged as author you would probaby have to submit a pull request.

[gerontec]

Thanks! Here it is: #991

It is rebased onto current master and contains two commits: the offset table is streamed to disk during the build (needed for nr on a 64 GB machine), and makedb --append. Build + append gives a .dmnd byte-identical to a single-step build, with and without taxonomy.