document Internals better #155

anarcat · 2014-12-16T14:12:54Z

this originally started as an attempt to document keyfiles, but after reading a mailing list discussion recommended in #27, turned in a fully-fledged attempt at documenting all internals, including chunking (#28).

one thing that seems to still be missing here is #45, the cache file.

this is still incomplete as it only describes key files, but doesn't clearly say how chunks are encrypted or decrypted. this address parts of jborg#29 but eventually that document should also cover jborg#27, jborg#28 and maybe jborg#45

this should address jborg#27, jborg#28 and jborg#29 at least at a basic level it is mostly based on the mailing list discussion mentionned in jborg#27, with some reformatting and merging of different posts.

again taken from the mailing list, mostly

anarcat · 2014-12-16T16:20:03Z

i believe the above error is not specific to this pull request.

anarcat · 2014-12-16T17:26:20Z

one thing that is missing here is the actual descriptions of all the fields, and an expanded description of the encryption process. the keyfile encryption process is clear, maybe it needs to be refactored with the encryption section as there are probably commonalities there.

anarcat · 2015-01-29T16:17:42Z

ping.

sotte · 2015-02-17T18:46:44Z

I was just starting to read the source when I thought that an "internals" page would be nice. This would be a nice addition to attic! Please merge the pull request.

anarcat · 2015-02-19T15:06:29Z

thanks for your support, @sotte - i think @jborg is not much around these days, but I hope he hears you. :)

ThomasWaldmann · 2015-03-01T00:25:37Z

please merge, see also:

from http://librelist.com/browser//attic/2014/1/31/attic-vs-obnam/ :

"""attic con: repository format not documented"""

@anarcat thanks for collecting / assembling / documenting this! It's very useful.

ThomasWaldmann · 2015-03-04T16:14:31Z

docs/internals.rst

+* time
+
+Each item represents a file or directory or
+symlink is stored as a ``item`` dictionnary that contains:


fixed in 8f8a035

ThomasWaldmann · 2015-03-04T16:42:17Z

docs/internals.rst

+When initialized with the ``init -e keyfile`` command, |project_name|
+needs an associated file in ``$HOME/.attic/keys`` to read and write
+the repository. The format is based on msgpack_, base64 encoding and
+PBKDF2_ SHA256 encryption, which is then encoded again in a msgpack_.


PBKDF2_ SHA256 key derivation function,

you're right, i changed that to use "hashing"

@ThomasWaldmann

this should fix the comments identified as `typo` and other small quirks found by @ThomasWaldmann.

@ThomasWaldmann

according to @ThomasWaldmann, the algorithm varies according to whether encryption is enabled.

i am actually assuming this right now, i haven't double-checked

it seems I extracted that data from [this mailing list post][] which in turn takes it from [this github comment][]. [this mailing list post]: http://librelist.com/browser/attic/2014/5/6/questions-and-suggestions-about-inner-working-of-attic/ [this github comment]: jborg#26 (comment)

do not use the word "encryption", as it is actually closer to "hashing" anyways.

anarcat · 2015-03-05T14:04:12Z

hi @ThomasWaldmann - thanks for the thorough review! I have done my best to incorporate changes that will reflect the concerns you raised, but there are two issues which I just cannot fix, and they are "why 2 keys" and "why 3 keys". :)

my gut feeling is that it is because you do not want to have one key (say the checksumming algorithm) be reused to encrypt the data to make the system more secure to (say) side-channel attacks. but I am not a cryptographer, so I am a little uncomfortable making such claims in the documentation.

furthermore, i would recommend you make the changes directly in your own branch - it would probably be easier for you than commenting here and for me than merging all those changes in by hand. :)

therefore, you may want to review the comments to make sure I covered all the cases you raised.

i sure hope this will be merged soon! :)

ThomasWaldmann · 2015-03-05T14:19:00Z

Sure, maybe @jborg can comment on the "key" questions. For the enc and enc_hmac key, as you I also suspect security reasons. I just stumbled over it because for aes-gcm (which I am working on right now), there would be only 1 key for both enc and mac.

Good idea about forking your docs branch. Maybe we could even create a "help with all the merging" branch somewhere in case @jborg is too busy otherwise to do merges now.

anarcat · 2015-03-05T14:45:00Z

re a separate repo/branch, i opened #217 to continue that conversation.

sa2ajj · 2015-03-09T19:50:13Z

May I suggest to keep documentation in 'one sentence per line format'? It drastically helps while reviewing changes.

This article has some background.

As for my original request -- describe how de-duplication works -- thank you very much!

ThomasWaldmann · 2015-03-09T19:56:22Z

@sa2ajj good idea if doc source is not read by end users.

paolodina · 2015-03-09T20:57:17Z

@sa2ajj one sentence per line format certainly has sense, but a similar result can be achieved without compromising the doc source readability using git.

git diff --word-diff=porcelain

Show a word diff, using the to delimit changed words.

porcelain
Use a special line-based format intended for script consumption.
Added/removed/unchanged runs are printed in the usual unified diff format, starting
with a +/-/ character at the beginning of the line and extending to the end of the
line. Newlines in the input are represented by a tilde ~ on a line of its own.

anarcat · 2015-03-09T22:01:42Z

why =porcelain, actually? i find the default word-diff=color much nicer...

paolodina · 2015-03-09T22:25:33Z

Another option, still better imo.

sa2ajj · 2015-03-18T18:11:44Z

FWIW, all those parameters to git diff do not really help when you review the change in GitHub.

However, as I said, it was just a suggestion.

anarcat · 2015-03-18T21:42:06Z

yeah, i can certainly appreciate the "one line per idea" concept, especially for those poor VCS that can handle complex things like paragraphs.

poor little github, it's not its fault it's proprietary.

document key files in a new Internals page

a8ab9c3

this is still incomplete as it only describes key files, but doesn't clearly say how chunks are encrypted or decrypted. this address parts of jborg#29 but eventually that document should also cover jborg#27, jborg#28 and maybe jborg#45

anarcat mentioned this pull request Dec 16, 2014

Documentation on how I used Attic Backup (full system backup) #60

Open

anarcat added 4 commits December 16, 2014 10:03

clarify some bits I missed

9f0ed2a

document more internals, based on mailing list discussion

3f27c36

this should address jborg#27, jborg#28 and jborg#29 at least at a basic level it is mostly based on the mailing list discussion mentionned in jborg#27, with some reformatting and merging of different posts.

document the repo config file and more storage properties

fd56bf0

again taken from the mailing list, mostly

add more details on how encryption works

1fde2a9

anarcat changed the title ~~document key files in a new Internals page~~ document Internals better Dec 16, 2014

anarcat added 5 commits December 16, 2014 10:30

add a more gentle introduction

ddca3b8

reorder to be more logical and more gentle

688ba10

fix reference errors and remove reference to source code

d58b6dd

fix formatting issues

b7c2673

better titles

e80e6c4

Update internals.rst

b7718f0

ThomasWaldmann reviewed Mar 4, 2015
View reviewed changes

ThomasWaldmann mentioned this pull request Mar 4, 2015

draw a picture from the internals documentation #216

Open

anarcat added 5 commits March 5, 2015 08:41

fix a bunch of typos

8f8a035

this should fix the comments identified as `typo` and other small quirks found by @ThomasWaldmann.

expand on the chunk id hash mechanism

87cb4a4

according to @ThomasWaldmann, the algorithm varies according to whether encryption is enabled.

clarify that 4095 bytes is not a typo

0ba8635

i am actually assuming this right now, i haven't double-checked

PDKF is a key derivation function

ecee5a0

do not use the word "encryption", as it is actually closer to "hashing" anyways.

This was referenced Mar 5, 2015

Document repository format #27

Open

Document how deduplication works #28

Open

Document how repository encryption works #29

Open

Document that hard links work #133

Open

helping out with merge reviews and issue triage #217

Open

maltefiala mentioned this pull request May 14, 2015

Dealing with attic issues borgbackup/borg#5

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

document Internals better #155

document Internals better #155

anarcat commented Dec 16, 2014

anarcat commented Dec 16, 2014

anarcat commented Dec 16, 2014

anarcat commented Jan 29, 2015

sotte commented Feb 17, 2015

anarcat commented Feb 19, 2015

ThomasWaldmann commented Mar 1, 2015

ThomasWaldmann Mar 4, 2015

anarcat Mar 5, 2015

ThomasWaldmann Mar 4, 2015

anarcat Mar 5, 2015

anarcat commented Mar 5, 2015

ThomasWaldmann commented Mar 5, 2015

anarcat commented Mar 5, 2015

sa2ajj commented Mar 9, 2015

ThomasWaldmann commented Mar 9, 2015

paolodina commented Mar 9, 2015

anarcat commented Mar 9, 2015

paolodina commented Mar 9, 2015

sa2ajj commented Mar 18, 2015

anarcat commented Mar 18, 2015

document Internals better #155

Are you sure you want to change the base?

document Internals better #155

Conversation

anarcat commented Dec 16, 2014

anarcat commented Dec 16, 2014

anarcat commented Dec 16, 2014

anarcat commented Jan 29, 2015

sotte commented Feb 17, 2015

anarcat commented Feb 19, 2015

ThomasWaldmann commented Mar 1, 2015

ThomasWaldmann Mar 4, 2015

Choose a reason for hiding this comment

anarcat Mar 5, 2015

Choose a reason for hiding this comment

ThomasWaldmann Mar 4, 2015

Choose a reason for hiding this comment

anarcat Mar 5, 2015

Choose a reason for hiding this comment

anarcat commented Mar 5, 2015

ThomasWaldmann commented Mar 5, 2015

anarcat commented Mar 5, 2015

sa2ajj commented Mar 9, 2015

ThomasWaldmann commented Mar 9, 2015

paolodina commented Mar 9, 2015

anarcat commented Mar 9, 2015

paolodina commented Mar 9, 2015

sa2ajj commented Mar 18, 2015

anarcat commented Mar 18, 2015