Remove saved configuration for devices not listed in sources anymore

[cr41gc]

I would like the ability to archive a device using the git output. I don't need to delete the device's history, but it would be nice to prevent the file from showing up when other users clone the backups repo.

I'd like the latest commit of the git repo to represent the state of the network, not just be an all-knowing backups location.

I've tried stopping oxidized, deleting device files from the git repo (not from oxidized's copy), then doing a git fetch -q origin master:master.

I tried to get around this by using the file output and doing git push commands myself, but then I found out the file output doesn't support type_as_directory, which I'm using to collect extra commands and put them into another folder.

Does anyone have a good way to archive or delete a backup file with the git output?

[devicenull]

We run this bash script weekly to cleanup the old devices... it'll probably need some updating for your environment:

#!/bin/bash


export OXIDIZED_PATH='/home/oxidized/.config/'

export WORKING_DIR='/tmp/oxidized_cleanup'

if [ `whoami` != 'oxidized' ]; then
        echo "Must run as oxidized user to avoid trashing repo permissions"
        exit 1
fi

rm -rf $WORKING_DIR
mkdir -p $WORKING_DIR
cd $WORKING_DIR
echo "Checking out repo"
git init-db
git remote add origin $OXIDIZED_PATH/oxidized/oxidized.git
git pull origin master

sudo service oxidized stop
echo "Evalulating files"
for filename in `git ls-tree --full-tree -r HEAD | awk '{print $4}' | grep -v \.xml`; do
        DEVICENAME=${filename//\.xml//}
        echo $DEVICENAME
        grep $DEVICENAME $OXIDIZED_PATH/router.db 2>&1 > /dev/null
        if [ $? -ne 0 ]; then
                echo "remove $filename"
                git rm -f $filename ${filename}.xml
        fi
done

echo "Comitting results"
git commit -a -m "Auto-purge"
git push origin master
sudo service oxidized start

[991jo]

Hello

I have the same issue. I would like to remove old devices from the HEAD of the git repo. It is okay that they remain in the git history, but we often use tools like "grep" on the config files to check stuff like "how many devices have config setting X". And when you have the old devices that no longer exist in the network in the repo you have false positives.

We have a script that cleans up the git repo, but the first commit that oxidized does adds all files that have been deleted.

the short version of what we are doing is:

systemctl stop oxidized
git clone oxidized.git temp_repo
cd temp_repo
git rm $old_router
git commit -m "removed $old_router"
git push
systemctl start oxidized

The first time oxidized finds a change and commits to the repo the $old_router is added to the repo again. I have no idea where oxidized even has this file. The routers that are deleted are not connected anymore and no longer in the router db.

[Nayruden]

I ran into the same problem @991jo saw. Hopefully, the explanation and workaround below will help others from creating head-shaped indentations in their walls.

The short story is that you'll want to run the following commands on the Oxidized repo after you make changes to the upstream (such as deletion).

cd /your/oxidized/repo.git
git fetch origin
git update-ref HEAD origin/master
rm index
git ls-tree -r HEAD | git update-index --index-info

The long story is that Oxidized is using a bare repo, but somehow leverages the Git index (aka cache or stage) even though it's not checked out (no working copy). This is not the usual way of doing things and thus the git commands most people are used to won't help you figure out why your deleted files keep getting pushed back in. The commands above purge the stale index and force it to be re-created using the latest commit upstream.

[lazynooblet]

This has been driving me insane having spent a couple of hours trying to get this to work properly. With the information in the comments above I just wasn't quite getting what I needed. In the end what worked was pushing the latest commit from the bare repo to upstream and then performing the steps outlined by Nayruden.

Steps taken:

# stop oxidized
docker stop oxidized
# clone bare repo
cd /tmp
git clone /path/to/oxidized/repo.git oxidized
# make changes
cd oxidized
git rm old-router
git commit -m "removed old-router"
git push
# oxidized bare git repo is now up to date, but we need to do more
# chdir to oxidized bare git repo folder
cd /path/to/oxidized/repo.git
# push the commit upstream
git push origin master
# update local repo as per notes by @Nayruden 
git fetch origin
git update-ref HEAD origin/master
rm index
git ls-tree -r HEAD | git update-index --index-info
# tidy cloned repo
rm -rf /tmp/oxidized
# start oxidized
docker start oxidized

[robertcheramy]

My understanding of this feature request:
Devices, which are no longer listed in the source (csv/http/sql) should be removed from the output (file/git/gitcrypt/http).

• For the file output this means deleting the file
• For the git output this means commiting a file delete
• For the http output I have no Idea ;-)

This deletion of the file should also be propagated by the githubrepo hook.

This might be not easy to realize, as oxidized currently does not save its state between restarts. When reloading the configuration, oxidized already detects a device ist not listed anymore, but this won't work for restarts.

I assign the feature request to myself, but will work on a very low priority & best effort on this. If someone wants to step in, feel free an tell us here.

[lazynooblet]

Having auto-delete is a nice-to-have but is tricky to implement. Having all files deleted because the config got messed would be an unwanted side effect.

Personally, I'd be content with the git implementation performing a pull prior to commit&push with auto-merge enabled. Fighting with the bare git repo and diagnosing how items were being cached was a complete 'mare.

I think everyone in this thread has been effected by trying to tidy up their git repo.

[robertcheramy]

Update (duplicating the information from my comment on #3121):

Oxidized writes an index to the repository, which is confusing as it is a bare repository.

The README.md of libgit2/rugged reads every time the tree from the index, and does not write the index to disk.

This explains why cloning the repository, making changes and pushing it back does not work:

• as we have a bare repository, git does not update the index
• oxidized still works with its old index, which have not been updated. So git commits on the old tree, updates HEAD there, and the changes are ignored.

A solution could be to reread the index from HEAD before every commit and do not store it to disk anymore (which may be a performance issue), or at least when starting oxidized (and still write the index to disk).
Another solution could be to have a working directory, which is not prefered by @ytti and would be a problem when upgrading from an oxidized version without working dir.