cmd/link, embed: store only one copy of an embedded []byte

[pjebs]

I am using the caire package: https://github.com/esimov/caire/blob/master/process.go#L26
It embeds a file called facefinder required for a common dependency called pigo.

My application also uses the pigo package and independently needs to embed the same file.

Currently the same file's data is being embedded twice. Perhaps as an optimisation, it can calculate the hashes of each embedded file (including inside all the dependencies), and store the data only once for duplicates.

This proposal is only for single file embeds and not directories.

[ianlancetaylor]

We can't do that for this example. When you use go:embed with a []byte, the []byte is initialized with the contents of the file. But the program can then change the contents of the []byte, as it can with any []byte value. So each separate go:embed requires a different []byte value.

[pjebs]

How about this?

The first []byte embed reference, it's stored as currently implemented. For all others, the app populates it with a new copy as part of the app loading process.

It's not a matter of doing it ourself in an func init() because we don't have access to what is going on in dependency packages.

Essentially this proposal is about reducing executable's file size in exchange for app start-up time - for cases where single files are duplicated.

[seankhliao]

This doesn't sound like a very common problem, and seems it would be better served by the dependency exposing the embedded data, either there or as a separate package, so the data doesn't get out of sync.

[mvdan]

As a reply to Ian: perhaps the deduplication could happen when embedded as a string. There is another advantage with strings for embedding already: they can live in read-only memory.

[randall77]

@mvdan I believe we already deduplicate when embedding strings.

[DeedleFake]

@randall77

Just ran a quick test. Yeah, it does. Running strings confirms that the string only appears in the file once, and printing the string headers shows identical data pointers.

Edit: Realized that I didn't test it with identical, but separate, files embedded in two separate packages, so I did that. Same result.

[pjebs]

I'm not sure if all the suggestions regarding deduplication already taking place for strings is a suggestion that the feature already exists if I choose to just embed as a string instead - OR if the suggestion is that it's already been done for strings and therefore for consistency, this feature should also be implemented for []byte as a tacit agreement of my proposal.

If it's the former, we have no control over what dependencies are doing internally. The proposal is about minimising executable file size.

[DeedleFake]

I believe that it's actually being used as evidence that it was deliberately not implemented for []byte.

[pjebs]

I have updated the proposal to treat string and []byte embeds as equivalent for deduplication purposes.

As such, all []byte embeds should be embedded as a string embed "behind-the-scenes", if the file hashes are the same (since deduplication is already supported), and then at app-start-up, copied to the relevant []byte reference.

[randall77]

It is not immediately clear to me that it is worth doing. You're basically trading off binary size for startup time and final memory usage.

If you embed the same []byte twice, you now have to copy that data twice on startup, and then you've got 3 copies of it in memory, at least until/if the original string copy gets uncached/paged out/whatever.

In the current scheme (embed it twice in the data section), then the binary is bigger but the startup time, and the ram use, is smaller.

@pjebs It sounds like you are more concerned about binary size than the other two, but maybe that's not true for everyone.

[pjebs]

@randall77 That's correct. I am more concerned with file size. Perhaps it's not true for everyone. Also bear in mind that go is supporting WASM, where on the browser, file size download is probably the bigger concern.

Also in the current scheme, it could be embedded multiple times > twice. If memory is the issue, under the proposal, the developer has control over clearing the []bytes if it's not truely necessary. In the current scheme, the developer has no control over minimising file size.