image layout: move to slash separation for blob storage layout

[stevvooe]

Currently, blobs are stored with the follow format:

blobs/<digest>-<hex bytes>

This has caused confusion in the context of the :-separated hash format.

To make this more clear, the blobs should be stored in the following format:

blobs/sha256/<hex bytes>

cc @vbatts @philips

[wking]

Or we could use <algo>-<hex-digest> for both the blob names and the blob filenames, in which case there would be even less confusion.

I'm ok with using a path separator, but Windows folks would have to be careful to use the appropriate separator for the engine (e.g. / for image-layout backed by HTTPS, tar, … and \ for image-layout backed by local directories).

[philips]

+1 to this concept.

[xiekeyang]

If it should add a subdirectory by highest 8 bit of hash digest, like blobs/sha256/1a/<hex bytes>?
This map can speed up retrieve, and avoid lots blob files in one folder.
In future, the storage will play a repository, so it will collect lots of blobs.
This way is widely used in CAS system.

[glestaris]

That would be nice. We can store multiple images under the same directory structure.

[stevvooe]

@xiekeyang @glestaris We have discussed the convention of using a portion of the digest to partition the CAS files but have been unable to find any recent examples of where this actually speeds up retrieval.

Do you have any data supporting this design? What is the actual performance impact?

[wking]

On Thu, Aug 25, 2016 at 07:06:12AM -0700, Stephen Day wrote:

@xiekeyang @glestaris We have discussed the convention of using a
portion of the digest to partition the CAS files but have been
unable to find any recent examples of where this actually speeds up
retrieval.

Previous discussion of hash sharding starting here 1.

[xiekeyang]

@stevvooe @wking Got it, thanks a lot!

[philips]

@runcom @s-urbaniak could you work-up a patch to make this change in the tooling? And then @stevvooe or I can make the english language change.

[stevvooe]

@philips I can take this on!

[philips]

@stevvooe are you going to do the code, english or both?

[philips]

@runcom volunteered to do the code side of this one.

[runcom]

created #230 for the code side