Request for feature: borg search

[mnikhil-git]

I was wondering if there is a borg search feature where given the list of files as input, I could get the list of backup archives which have the files in them.

[ThomasWaldmann]

borg does not have an index, so there is nothing to search in.

Except borg list / captured borg create --list output, which you can do yourself (e.g. using grep).

I made some experiments a while ago using whoosh for indexing/searching, but it created huge index files (for huge archives), these always needed to be updated for new archives, ... - too much time and space usage. So it didn't continue work in that direction.

[mnikhil-git]

Yeah, I was hoping that instead of me wrapping around the list of borg list, borg info and then list of each archive and then grep for the list of files in each archive, it would be good to have borg do it since it has access to the methods already.

[ThomasWaldmann]

if you need this frequently, i'ld suggest doing the backups with borg create --list ... >archive-xxxxx.list 2>&1 (and store the listings at a safe place) - then you can quickly grep the already existing listings.

if you need it infrequently, you can just ad-hoc use borg list to generate the listing(s) as needed.

implementing grep into borg would be somehow against unix philosophy.

also, when implementing it so it searches all archives, it would be extremely slow if one has many and/or big archives.

[mnikhil-git]

implementing grep into borg would be somehow against unix philosophy.

hmm..this is like not straight grep but a recursive one across all the available archives for a list of files to check for their inclusion and I think it would be better to let borg do that since it can iterate over the list of archives and the list of files for each of them and do a filter search.

[talbym]

I was also searching for a search function, expecting I could receive a listing like the "borg list" command in conjunction with a "grep" command produces it IF I know already where to search and only then can grep for it. It would really be nice if borg could provide me a tool by which I could locate available versions of a file or dir, for the case I would have lost it, or having edited it in bad way!
If this tool would simply be a some shell script, which then searches through all my Repos and Archives, then this would be enough - would be nice to find such example script in the documentation so that I could copy and use it, instead of myself first having to learn how to program at the necessary skill level.

borg search MyFileName MyOtherFileName SomeDirName
MyFileName
from DateOfBackup in Repo1::Archive3
-rw-r--r-- me me 464 Sat, 2018-04-28 12:38:02 home/me/TheDir/MyFileName
from DateOfBackup in Repo1::Archive2
-rw-r--r-- me me 200 Sat, 2018-04-28 12:38:02 home/me/TheDir/MyFileName
from DateOfBackup in Repo1::Archive1
-rw-r--r-- me me 10 Sat, 2018-04-28 12:38:02 home/me/TheDir/MyFileName
from DateOfBackup in Repo7::Archive6
-rw-r--r-- me me 999 Sat, 2018-04-28 12:38:02 anotherStorage/someDir/MyFileName

MyOtherFileName
.
.
.

SomeDirName
.
.
.

[ThomasWaldmann]

borg mount -o versions ... repo ... might be also an option to easily locate some specific file version.

[ThomasWaldmann]

guess this is solved.

[EHJ-52n]

I would disagree and see this issue not as solved. I have the use case like others mentioned before, that I do not know when a file was "backuped". Hence, I want to search in all available backups of my repo for a pattern, file or directory name.

backup2l, for instance, being a very old backup solution, offers you a simple command to --locate all backup locations for a given pattern.

What can I do that a search command is added to borg backup?

[henfri]

Hello,

I have the same issue (searching for a file in the repo).
What I tried sofar:
1) borg list path/to/repo |grep filename -- lists only the archives, but not the files in the archive
2) borg mount path/to/repo /mnt/borg && find /mnt/borg/ -name filename -- after a while (an hour or so) I get error messages (sorry, I have not recorded, but I can reproduce if needed); also after this while the system gets very (!) slow
3) borg mount -o versions -- after a while it only says "killed" on the commandline
4) borg mount path/to/repo /mnt/borg && ls homeserver-documents-2022-*/srv/Daten/ leads to:
ls: Zugriff auf 'homeserver-documents-2022-*/srv/Daten/' nicht möglich: Der Socket ist nicht verbunden (I think this is also the error I got in (2) and a load avg of 40(!)
Edit: I had coincidentially fatrace running in parallel (on a different mountpoint though) and it showed: fatrace: open_by_handle_at: Too many levels of symbolic links

Do you have any further Ideas?

Best regards,
Hendrik

[ThomasWaldmann]

1. you can create a list of all archives with some scripting: borg list --short REPO then iterate through that list and call borg list REPO:ARCHIVE to get a files listing of all archives.

2. would be interesting if it is not just "out of memory"

3. likely the OOM killer killing the borg process because it used up all your RAM

4. not sure about this one

Guess you first need to fix the memory issue. Either upgrade memory or use less memory. borg mount has some options to only partially make stuff available (like only processing some archives or specific paths).

[hokascha]

I use this script to search for a pattern in all archives:

#!/bin/bash
export BORG_PASSPHRASE="<password>"
borg list --short  <REPO> | while read -r a ; do
    echo "$a"
    borg list <REPO>::$a | grep "$1"
done

save as search.sh, chmod +x search.sh and ./search.sh <pattern> to search.

[EricDuminil]

@hokascha Excellent, thanks! It's very concise, convenient, and worked fine.

If anyone's interested, I wrote a somewhat longer script, with parameters, which allows to use env files or set BORG_REPO:

#!/bin/sh

########################
#      BORG GREP       #
########################

# Search for a pattern in a Borg repository
# When was the last or first time a file was included in a Borg archive?
# Inspired by https://github.com/borgbackup/borg/issues/4092#issuecomment-1403696981

usage() {
    echo "Usage: $0 <pattern> [--repo <repo>] [--env <env_file>]"
    echo "  pattern:     Pattern to search for in archive contents"
    echo "  --repo:      Borg repository path (optional if BORG_REPO is set)"
    echo "  --env:       Environment file to source (default: .env if it exists)"
    echo ""
    echo "Examples:"
    echo "  $0 '*.log'"
    echo "  $0 'folder/subfolder/specific_file.txt'"
    echo "  $0 '*.log' --repo /path/to/repo"
    echo "  $0 '*.log' --env custom.env"
    echo "  $0 '*.log' --repo /path/to/repo --env custom.env"
    exit 1
}

# Check if pattern is provided
if [ $# -lt 1 ]; then
    usage
fi

PATTERN="$1"
shift

# Set default env file
ENV_FILE=".env"

# Parse named parameters
while [ $# -gt 0 ]; do
    case "$1" in
        --repo)
            if [ $# -lt 2 ]; then
                echo "Error: --repo requires a value"
                exit 1
            fi
            REPO="$2"
            shift 2
            ;;
        --env)
            if [ $# -lt 2 ]; then
                echo "Error: --env requires a value"
                exit 1
            fi
            ENV_FILE="$2"
            shift 2
            ;;
        *)
            echo "Error: Unknown parameter '$1'"
            usage
            ;;
    esac
done

# Source environment file if it exists
if [ -f "$ENV_FILE" ]; then
    echo "Sourcing $ENV_FILE..."
    # Use . instead of source for POSIX compatibility
    . "./$ENV_FILE"
fi

# Set repository from parameter if provided, otherwise use environment variable
if [ -n "$REPO" ]; then
    BORG_REPO="$REPO"
fi

# Check if BORG_REPO is set
if [ -z "$BORG_REPO" ]; then
    echo "Error: BORG_REPO not specified as parameter or environment variable"
    exit 1
fi

# Check if BORG_PASSPHRASE is set, if not ask for it
if [ -z "$BORG_PASSPHRASE" ]; then
    printf "Enter Borg passphrase: "
    # Disable echo for password input
    stty -echo
    read BORG_PASSPHRASE
    stty echo
    echo
fi

# Export BORG_REPO and BORG_PASSPHRASE for borg commands
export BORG_REPO
export BORG_PASSPHRASE

echo "Searching for pattern '$PATTERN' in repository: $BORG_REPO"
echo

# Get list of archives and search through each one
borg list --short | while IFS= read -r archive; do
    echo "Archive: $archive"
    borg list ::$archive | grep "$PATTERN"
    echo
done

This way, BORG_REPO or PASSPHRASE don't have to be hardcoded in the script.

❯ ./borg_grep.sh
Usage: ./borg_grep.sh <pattern> [--repo <repo>] [--env <env_file>]
  pattern:     Pattern to search for in archive contents
  --repo:      Borg repository path (optional if BORG_REPO is set)
  --env:       Environment file to source (default: .env if it exists)

Examples:
  ./borg_grep.sh '*.log'
  ./borg_grep.sh 'folder/subfolder/specific_file.txt'
  ./borg_grep.sh '*.log' --repo /path/to/repo
  ./borg_grep.sh '*.log' --env custom.env
  ./borg_grep.sh '*.log' --repo /path/to/repo --env custom.env