1.1.0 - nodump flag excluding stuff on unraid

[airbillion]

I have been using borg for well over a year now...I have updated between versions everytime without any issues except for the latest release...1.1.0.

When I run my backups it seems as though borg is not seeing all the files in my directories...I have tried all the different options using the --files-cache without any success...when I mount an archive after running a backup only a few directories are shown..if i downgrade back to 1.0.11 and run the backup again my whole directory is backed up correctly...

The report after running shows:

borg 1.1.0:
------------------------------------------------------------------------------                                                                                      
Archive name: Tower-DOWNLOADS-2017-10-10
Archive fingerprint: 9866d6dd574c6488f2bbfdbf8003695556da0acb690b71a5eedf756b0e224442
Time (start): Tue, 2017-10-10 13:54:28
Time (end):   Tue, 2017-10-10 13:54:28
Duration: 0.14 seconds
Number of files: 0
Utilization of max. archive size: 0%
------------------------------------------------------------------------------
                       Original size      Compressed size    Deduplicated size
This archive:                  631 B                585 B                585 B
All archives:                1.00 TB            879.78 GB            106.85 GB

                       Unique chunks         Total chunks
Chunk index:                   76804               621866
------------------------------------------------------------------------------
Keeping archive: Tower-DOWNLOADS-2017-10-10           Tue, 2017-10-10 13:54:28 [9866d6dd574c6488f2bbfdbf8003695556da0acb690b71a5eedf756b0e224442]
Keeping archive: Tower-DOWNLOADS-2017-10-09           Mon, 2017-10-09 00:15:55 [463b52fffc51b982037df5464310d70c4288c58e99d657ede6de248fa2aeb9e8]
Keeping archive: Tower-DOWNLOADS-2017-10-08           Sun, 2017-10-08 00:15:38 [e027fb3f1bf2653e8744107382e7ae09933ac04fb7df7ee2b9a570ab0b58f3cb]
Keeping archive: Tower-DOWNLOADS-2017-10-07           Sat, 2017-10-07 00:15:43 [43663b684f99127e7e2e991cc0f04478746c36a1eb23693fa1872bf4f8d40ea3]
Keeping archive: Tower-DOWNLOADS-2017-10-06           Fri, 2017-10-06 00:15:59 [15124fb778a0aac88e0077dc5f3720c351d18223f59b72b52fff53a012ca6966]
Keeping archive: Tower-DOWNLOADS-2017-10-05           Thu, 2017-10-05 00:15:36 [aab098fcb86d6f290fd45c1a48020e28f4f4785a9e7e44887fc197955f7c63a5]
Keeping archive: Tower-DOWNLOADS-2017-10-04           Wed, 2017-10-04 00:15:59 [77471d3464a2029788f0a2e66c420769dd08ff22ea5dbb303baf5db96a2727b1]
Keeping archive: Tower-DOWNLOADS-2017-10-03           Tue, 2017-10-03 00:16:12 [c1b20304b7a39fef4b556db4c2ffeefff52b7bad12d43c105f84a8bee50754cd]
------------------------------------------------------------------------------


borg 1.0.11
------------------------------------------------------------------------------                                                                                      
Archive name: Tower-DOWNLOADS-2017-10-10
Archive fingerprint: f08500be0803025d671f11a224215179635ff2bf83a6e42fa4d6302b768ecba3
Time (start): Tue, 2017-10-10 14:31:20
Time (end):   Tue, 2017-10-10 14:32:05
Duration: 44.78 seconds
Number of files: 40352
------------------------------------------------------------------------------
                       Original size      Compressed size    Deduplicated size
This archive:              148.08 GB            130.39 GB            480.64 kB
All archives:                1.15 TB              1.01 TB            106.86 GB

                       Unique chunks         Total chunks
Chunk index:                   76809               712568
------------------------------------------------------------------------------
Keeping archive: Tower-DOWNLOADS-2017-10-10           Tue, 2017-10-10 14:31:20
Keeping archive: Tower-DOWNLOADS-2017-10-09           Mon, 2017-10-09 00:15:55
Keeping archive: Tower-DOWNLOADS-2017-10-08           Sun, 2017-10-08 00:15:38
Keeping archive: Tower-DOWNLOADS-2017-10-07           Sat, 2017-10-07 00:15:43
Keeping archive: Tower-DOWNLOADS-2017-10-06           Fri, 2017-10-06 00:15:59
Keeping archive: Tower-DOWNLOADS-2017-10-05           Thu, 2017-10-05 00:15:36
Keeping archive: Tower-DOWNLOADS-2017-10-04           Wed, 2017-10-04 00:15:59
Keeping archive: Tower-DOWNLOADS-2017-10-03           Tue, 2017-10-03 00:16:12
------------------------------------------------------------------------------
                       Original size      Compressed size    Deduplicated size
Deleted data:                    0 B                  0 B                  0 B
All archives:                1.15 TB              1.01 TB            106.86 GB

                       Unique chunks         Total chunks
Chunk index:                   76809               712568
------------------------------------------------------------------------------

Is there anything I am missing that I should do befor I upgrade to 1.1.0?

Any help would be appreciated!!

Thanks

[ThomasWaldmann]

Can you add the full borg command you use? Also, if we talk about borg create, remove the borg prune output please.

[airbillion]

Sure...sorry....

command:

'borg create -v -s -p --list --filter=AME --files-cache=ctime,size --compression lz4' "root@192.168.1.135:/mnt/user/borg_backup_repos/downloads/"::"$(hostname)-DOWNLOADS-$now" "/mnt/user/downloads/"

[ThomasWaldmann]

can you try without the -p --filter? can you catch all the output, stdout and stderr?
also, remove the trailing slashes from repo and source path?
the double quotes everywhere seem also not really needed. borg can do hostname and now by itself.

[airbillion]

Hi,

Sure... everything is in quotes because I pulled it all from my backup script which uses variables...

I use a script that has been running nightly for over a year now...I also created a test repo with a small test folder to backup and have the same issues...

Using a script helps with my export values for the location of my backup repo and locations for the cache...

I'll post my results here later...thank you for the help!!

I hope I can get this figured out...

[enkore]

Check if there is a NODUMP flag on the files or directories (lsattr(1) shows a "d").

[airbillion]

Test repo creation with -P & --filter removed...trailing slashes removed from repo and backup directory...quotes also removed:

root@Tower:/boot/scripts/borg_scripts/repo_scripts# ./borg_test.sh 

Borg TEST Backup script started....Tue Oct 10 22:27:32 EDT 2017
x /mnt/user/downloads/Books/walking_dead
------------------------------------------------------------------------------
Archive name: Tower-TEST-2017-10-10-1507688852
Archive fingerprint: b2edbd38c7a2db804260497979c34bc15c911b31e1e476f014972343baed77b0
Time (start): Tue, 2017-10-10 22:27:35
Time (end):   Tue, 2017-10-10 22:27:35
Duration: 0.00 seconds
Number of files: 0
Utilization of max. archive size: 0%
------------------------------------------------------------------------------
                       Original size      Compressed size    Deduplicated size
This archive:                  537 B                532 B                532 B
All archives:              686.57 MB            649.65 MB            575.44 MB

                       Unique chunks         Total chunks
Chunk index:                     824                  897
------------------------------------------------------------------------------

Looks like it is excluding the whole directory for some reason...I have no exclude in the create command...

[ThomasWaldmann]

Is this backing up a mounted smb share exported by macOS?

Please check what @enkore mentioned, it is likely the root cause for this behavior: #3127 (comment)

[airbillion]

no....a local share...
Very odd because when I revert to version 1.0.11...the whole share backs up normally..

[airbillion]

How do I check for a NODUMP flag on the files or directories (lsattr(1) shows a "d")?

I am not using a MacOS machine...

Here is the command:

borg create -v -s --list --compression lz4 root@192.168.1.135:/mnt/user/borg_backup_repos/test::$(hostname)-TEST-$now /mnt/user/downloads/Books/walking_dead

[airbillion]

lsattr output:

root@Tower:/boot/scripts/borg_scripts/repo_scripts# cd /mnt/user/downloads/Books/walking_dead/
root@Tower:/mnt/user/downloads/Books/walking_dead# lsattr
------dA----tTe--N- ./The Walking Dead 144 (2015) (Digital) (Minutemen-Acan).cbr
-----a---Ej-------P ./The.Chilling.Archives.of.Horror.Comics!.005.-.Haunted.Horror.2013.digital-Empire
---D------jIt--h-N- ./The.Walking.Dead.Compendium.V01.2011.Digital-EMPIRE
---------E-It--h--P ./The.Walking.Dead.Compendium.V01.2011.Digital-EMPIRE.1
---Dia---EjI------- ./The.Walking.Dead.Graphic.Novels
-----a-A---I------- ./The.Walking.Dead.Compendium.V01.2011.Digital-EMPIRE.2
-----a---Ej-------P ./The Walking Dead 149 (2015) (Digital) (Zone-Empire)
---D------jIt--h-N- ./The Walking Dead 150 (2016) (Digital) (Zone-Empire)
------dA----tTe--N- ./The Walking Dead 151 (2016) (Digital) (Zone-Empire)
---Di-d------T---N- ./The Walking Dead 152 (2016) (Digital) (Zone-Empire)
-----a-A---I------- ./The Walking Dead 153 (2016) (Digital) (Zone-Empire)
-----a---Ej-------P ./The Walking Dead 154 (2016) (Digital) (Zone-Empire)
-----ad---jI-T----P ./The Walking Dead 155 (2016) (Digital) (Zone-Empire)
------d--EjI------- ./The Walking Dead 156 (2016) (Digital) (Zone-Empire)
---Di-d------T---N- ./The Walking Dead 157 (2016) (Digital) (Zone-Empire)
---D---A-E--t--h-N- ./The Walking Dead 158 (2016) (Digital) (Zone-Empire)
---------E-It--h--P ./The Walking Dead 159 (2016) (Digital) (Zone-Empire)
-----ad---jI-T----P ./The.Walking.Dead.160.2016.Digital.Zone-Empire

[ThomasWaldmann]

don't do the lsattr in that dir, but one level higher.
you see at the x that borg is not even entering that directory.

the bsdflags support (includes detecting the NODUMP flag saying "do not backup this", dump is a backup tool) is new since borg 1.1.

[airbillion]

Output:

root@Tower:/mnt/user/downloads/Books# lsattr

-----ad--E--tT-h--P ./!GMA SCAN
---D-adA----t-e--N- ./walking_dead
----i-dA-E--t--h--- ./ebooks

[airbillion]

Was something changed between 1.1.0 and 1.0.11 that would cause difference?

I just reverted back to the 1.0.11 binary and the backup was made successfully withoutexcluding the directory...

[ThomasWaldmann]

Yes, I already said: 1.1 added bsdflags support (including NODUMP flag).

The flags on your files / dirs look strange, almost random.

https://linux.die.net/man/1/chattr see Attributes section there for an explanation of the flags.

What are the software levels below borg. What's "uraid"? What filesystem is that?

[airbillion]

Sorry...i didn't catch that...must be part of the issue

Unraid is a NAS OS based on linux slackware: https://lime-technology.com/what-is-unraid/

It is very popular and has been working exellent with borg for quite a few people...

Filesysem is xfs...my drives are pooled into a large storage array using fuse