picked cherries for 1.0-maint #2284

enkore · 2017-03-10T18:08:21Z

Changes from

#2156
#2159
#2171
#2181
#2187
#2198
#2206
#2224
#2254

# Conflicts: # docs/development.rst

# Conflicts: # borg/archiver.py

we do not trust the remote, so we are careful unpacking its responses. the remote could return manipulated msgpack data that announces e.g. a huge array or map or string. the local would then need to allocate huge amounts of RAM in expectation of that data (no matter whether really that much is coming or not). by using limits in the Unpacker, a ValueError will be raised if unexpected amounts of data shall get unpacked. memory DoS will be avoided. # Conflicts: # borg/archiver.py # src/borg/archive.py # src/borg/remote.py # src/borg/repository.py

borgbackup#2180 # Conflicts: # borg/archive.py

also: add some missing assertion messages severity: - no issue on little-endian platforms (== most, including x86/x64) - harmless even on big-endian as long as refcount is below 0xfffbffff, which is very likely always the case in practice anyway.

* trigger bug in --verify-data, see borgbackup#2221 * raise decompression errors as DecompressionError, fixes borgbackup#2221 this is a subclass of IntegrityError, so borg check --verify-data works correctly if the decompressor stumbles over corrupted data before the plaintext gets verified (in a unencrypted repository, otherwise the MAC check would fail first). * fixup: fix exception docstring, add placeholder, change wording

codecov-io · 2017-03-10T19:19:52Z

Codecov Report

Merging #2284 into 1.0-maint will decrease coverage by 0.15%.
The diff coverage is 58.82%.

@@             Coverage Diff              @@
##           1.0-maint   #2284      +/-   ##
============================================
- Coverage      81.96%   81.8%   -0.16%     
============================================
  Files             15      15              
  Lines           5172    5210      +38     
  Branches         880     887       +7     
============================================
+ Hits            4239    4262      +23     
- Misses           676     687      +11     
- Partials         257     261       +4

Impacted Files	Coverage Δ
borg/repository.py	`87.87% <100%> (+0.01%)`	✅
borg/archive.py	`80.34% <37.03%> (-1.17%)`	❌
borg/archiver.py	`81.28% <60.56%> (+0.52%)`	✅
borg/remote.py	`78.15% <80%> (-0.08%)`	❌
borg/helpers.py	`87.8% <80%> (-0.13%)`	❌
borg/locking.py	`89.58% <0%> (-1.05%)`	❌

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5212175...be1227f. Read the comment docs.

ThomasWaldmann · 2017-03-10T20:40:05Z

borg/archive.py

+                if e.errno == errno.E2BIG:
+                    logger.warning('%s: Value or key of extended attribute %s is too big for this filesystem' %
+                                   (path, k.decode()))
+                elif e.errno not in (errno.ENOTSUP, errno.EACCES):


this is a bit strange now: warns if we could not extract xattrs (because they were too big), but says nothing if we could not extract at all (because not supported or not permitted). this was fixed in master.

You need to cherrypick #2258 for this.

ThomasWaldmann · 2017-03-10T20:47:27Z

borg/helpers.py

    if hashindex.API_VERSION != '1.0_01':
        raise ExtensionModuleError
    if chunker.API_VERSION != '1.0_01':
        raise ExtensionModuleError
+    if compress.API_VERSION != '1.0_01':
+        raise ExtensionModuleError


oh? was this missing still in 1.0?

ThomasWaldmann · 2017-03-10T20:49:51Z

borg/remote.py

+    elif kind == 'client':
+        args.update(dict(max_array_len=LIST_SCAN_LIMIT,  # result list from repo.list() / .scan()
+                         max_map_len=100,  # misc. result dicts
+                         ))


these might be not the tightest limits as 1.0 does not use the dict-based rpc protocol, but should work I guess.

ThomasWaldmann · 2017-03-10T20:51:33Z

borg/testsuite/archiver.py

@@ -245,7 +245,7 @@ def cmd(self, *args, **kw):
        return output

    def create_src_archive(self, name):
-        self.cmd('create', self.repository_location + '::' + name, src_dir)
+        self.cmd('create', '--compression=lz4', self.repository_location + '::' + name, src_dir)


hmm, do we want that in 1.0?

Doesn't make much of a difference, but might be good to have when backporting new tests that expect it.

enkore · 2017-03-10T21:34:39Z

Okay, so I'm going to pick up #2199 and #2258 as well

(cherry picked from commit 4b33c3f)

…borgbackup#2258) * Set warning exit code when xattr is too big * Warnings for more extended attributes errors (ENOTSUP, EACCES) * Add tests for all xattr warnings (cherry picked from commit 63b5cbf)

enkore · 2017-03-10T21:47:01Z

And #2184 (but neither #2204 nor #2255, correct)?

ThomasWaldmann · 2017-03-10T21:56:17Z

@enkore yup. let's test the fixed async processing in master / 1.1b first for a while, then we can think whether we want it in 1.0 also.

borgbackup#1975 (cherry picked from commit 4d81b18)

enkore · 2017-03-10T22:04:09Z

Done

enkore · 2017-03-10T22:05:52Z

borg/archiver.py

+                # note: might crash in compact() after committing the repo
+                repository.commit()
+                logger.info('Done. Run "borg check --repair" to clean up the mess.')
+                return self.exit_code


I changed the error reporting here to match the the singular-behaviour that borg delete normally has in 1.0.x (RC=2 if archive not found, no plural messages etc.)

ThomasWaldmann

lgtm.

for easier review, let's merge this and start a new PR for further stuff.

enkore and others added 10 commits March 10, 2017 18:30

new branching model

9cb4b96

# Conflicts: # docs/development.rst

create: handle BackupOSError on a per-path level in one spot

26b4a39

# Conflicts: # borg/archiver.py

archive check: detect and fix missing all-zero replacement chunks, fixes

6e27c8b

borgbackup#2180 # Conflicts: # borg/archive.py

Print a warning for too big extended attributes

d11137d

Add testsuite to test handling of too big xattr

4543691

Location: accept //servername/share/path

391e302

Docs: one link per distro in the installation page

008c236

enkore changed the title ~~picked cherries~~ picked cherries for 1.0-maint Mar 10, 2017

enkore modified the milestones: 1.0.11, 1.0.11rc1 Mar 10, 2017

ThomasWaldmann reviewed Mar 10, 2017

View reviewed changes

Abogical added 2 commits March 10, 2017 22:44

Add return code functions (borgbackup#2199)

b4c0dcf

(cherry picked from commit 4b33c3f)

extract: warning RC for unextracted big extended attributes, followup (…

bfc7452

…borgbackup#2258) * Set warning exit code when xattr is too big * Warnings for more extended attributes errors (ENOTSUP, EACCES) * Add tests for all xattr warnings (cherry picked from commit 63b5cbf)

borg delete --force --force to delete severely corrupted archives, fixes

be1227f

borgbackup#1975 (cherry picked from commit 4d81b18)

enkore commented Mar 10, 2017

View reviewed changes

ThomasWaldmann approved these changes Mar 11, 2017

View reviewed changes

enkore merged commit 13455ab into borgbackup:1.0-maint Mar 11, 2017

enkore deleted the cp1 branch March 11, 2017 13:43

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

picked cherries for 1.0-maint #2284

picked cherries for 1.0-maint #2284

enkore commented Mar 10, 2017 •

edited

codecov-io commented Mar 10, 2017 •

edited

ThomasWaldmann Mar 10, 2017

Abogical Mar 10, 2017

ThomasWaldmann Mar 10, 2017

enkore Mar 10, 2017

ThomasWaldmann Mar 10, 2017 •

edited

ThomasWaldmann Mar 10, 2017

enkore Mar 10, 2017 •

edited

enkore commented Mar 10, 2017

enkore commented Mar 10, 2017

ThomasWaldmann commented Mar 10, 2017

enkore commented Mar 10, 2017

enkore Mar 10, 2017

ThomasWaldmann left a comment

picked cherries for 1.0-maint #2284

picked cherries for 1.0-maint #2284

Conversation

enkore commented Mar 10, 2017 • edited

codecov-io commented Mar 10, 2017 • edited

Codecov Report

ThomasWaldmann Mar 10, 2017

Choose a reason for hiding this comment

Abogical Mar 10, 2017

Choose a reason for hiding this comment

ThomasWaldmann Mar 10, 2017

Choose a reason for hiding this comment

enkore Mar 10, 2017

Choose a reason for hiding this comment

ThomasWaldmann Mar 10, 2017 • edited

Choose a reason for hiding this comment

ThomasWaldmann Mar 10, 2017

Choose a reason for hiding this comment

enkore Mar 10, 2017 • edited

Choose a reason for hiding this comment

enkore commented Mar 10, 2017

enkore commented Mar 10, 2017

ThomasWaldmann commented Mar 10, 2017

enkore commented Mar 10, 2017

enkore Mar 10, 2017

Choose a reason for hiding this comment

ThomasWaldmann left a comment

Choose a reason for hiding this comment

enkore commented Mar 10, 2017 •

edited

codecov-io commented Mar 10, 2017 •

edited

ThomasWaldmann Mar 10, 2017 •

edited

enkore Mar 10, 2017 •

edited