osutil: generalize SyncDir with FileState interface

[zyga]

The directory synchronization code grew out of the desire to have a set
of files described by a glob and short, in-memory contents be reflected
to the disk in an efficient and predictable way.

Recently this code has started to be used to install icon themes shipped
by snaps. This means it may be used to coerce snapd to read arbitrary
amount of data into memory.

This address this issue by generalizing the directory sync APIs to take
an interface instead of a concrete representation of the desired file.

There are now three concrete implementations, one that keeps the content
in memory, just like before, called MemoryFileState and two new ones:
FileReference and FileReferencePlusMode. Those both refer to an existing
file for content, opening up the possibility to refer to large files.
They only differ in the treatment of file mode, either mirroring the
mode of the file being referred or using a fixed mode, respectively.

Behind the scenes the EnsureFileState function will no longer read all
of the file into memory. Instead if will use FileReference to stream it,
chunk by chunk, in an attempt to see if the file is identical to what we
expected.

On top of that, if the file is not the same and the caller has provided
a FileReference or FileReferencePlusMode, the logic that writes a new
file and replaces the original is also using streaming, again saving a
in-memory copy.

This way we can now process files of arbitrary size using fixed amount
of memory. This involves the new icon wrapper which has been switched to
use FileReferencePlusMode.

The patch contains some verbose automatic changes around the code using
maps of FileState structure to replace them with maps of FileState
interface instead.

Signed-off-by: Zygmunt Krynicki me@zygoon.pl

[zyga]

@bboozzoo thank you for the review. I'll look at addressing this after the standup.

[pedronis]

couple of naming comments, I haven't looked myself at the equality code

[zyga]

I've addressed everything related to stream equality functions. I also applied the new suggested names.

[bboozzoo]

or bytes.NewBufferString(text)

[chipaca]

or strings.NewReader(text). We're spoiled for choice :-)

[zyga]

Or... what I did? :D

[chipaca]

or strings.NewReader(text). We're spoiled for choice :-)

[zyga]

Per @mvo5's request I'll split this into two branches:

• make FileState an interface, introducing MemoryFileState as a replacement
• the rest of the changes, unless they are absolutely required by the change above

EDIT: After brief discussion on IRC I'll refrain from splitting this.

[bboozzoo]

LGTM