Please clarify key names and new SLEEP file types

[bnewbold]

I recently created a dat archive:

bnewbold@orithena$ dat -v
13.8.1
bnewbold@orithena$ tree .dat/
.dat/
├── content.bitfield
├── content.key
├── content.secret_key
├── content.signatures
├── content.tree
├── metadata.bitfield
├── metadata.data
├── metadata.key
├── metadata.latest
├── metadata.ogd
├── metadata.signatures
└── metadata.tree

First, I notice that content.secret_key is in there, but metadata.secret_key. I have heard (and previously read somewhere) that secret keys live in ~/.dat/secret_key/, and indeed I see keys in there. @maxogden on IRC seemed concerned that content.secret_key was not in the homedir folder, and indeed, I believe the metadata register does not include any hashes or signatures of the content register (only index and length), so an attacker could inject corrupted/modified data. To be clear, this isn't any kind of remote exploit weakness, it only applies to the case where somebody copies a whole .dat directory around, copying and potentially exposing key material.

The metadata.latest and metadata.ogd files are not mentioned in the whitepaper. On IRC @maxogden said that metadata.ogd (:wink: :wink: ) mean "this is the original dat folder", which I assume is supposed to be a flag that the metadata secret key should be available and the dat overall is writable... is this the canonical flag for this, or should compatible clients also look in ~/.dat/secret_keys/ on their own? What is metadata.latest for?

If somebody replies to this issue, I can send a PR to include answers in the whitepaper.

[mafintosh]

There was a bug where the content secret key got written. That is fixed in latest (it's an empty file now)

[bnewbold]

What about metadata.latest and metadata.ogd?

[bnewbold]

Secret keys end up in ~/.dat/secret_keys/<shard>/<discovery_key>, where discovery key is in hex; the first two hex characters are the shard, and the rest of the key is the file name (so <discovery_key> is only 62 hex characters). Contents are just binary (64 bytes). This is an unstable API... not in the whitepaper, potentially application-specific.

metadata.ogd means this is the "OriGinal Dat", implying that this was the device and folder used to initialize the archive, and that a secret key should be found in the above location (on this device). This is an unstable API.

metadata.latest is a pointer to how far hyperdrive got during the last sync; this is a uint64 in big-endian byte order (so, an 8 byte file). This is an unstable API.

I have a lot of empty secret key files (just the discovery key file name)... probably a bug with opening files write/append. TODO: file a bug for this.