-
Notifications
You must be signed in to change notification settings - Fork 17.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
embed, cmd/go: add support for embedded files #41191
Comments
Would it be an error to have a |
@jimmyfrasche Yes, fifth-to-last bullet in list at https://go.googlesource.com/proposal/+/master/design/draft-embed.md#go_embed-directives. |
@rsc Maybe I have missed it in the draft, but I don't see the ability to embed a single file that you mention in your comment. |
@pierrec It's not in the draft doc (the "one addition" is the text in the comment above). Const strings can end up playing a role in deciding whether a program type checks, which would mean all type checkers would need to understand //go:embed'ed consts. In contrast, if we stick to vars, type checkers are none the wiser and can be left alone. Seems like we should probably stick to vars. Is there a particular reason you wanted a const instead of a var? Using them should be about the same as far as efficiency. (References to const strings end up compiling to what amount to references to hidden vars anyway.) |
Thanks for the explanation. I tend to embed static assets as const strings at the moment this is why I asked. I am fine with vars too! |
Interesting, so I could do something like: //go:embed version.txt
var Version string And potentially even have a |
Is it an error if the file isn't found? If so, where technically is the error considered to occur? Link time? |
Can we make sure that go:embed runs after go:generate so we can do things like easily generate versions, etc? |
From my understanding, |
@carlmjohnson Yes, it is always an error to say |
I think that this proposal is not complete without saying something about ETag. |
@tv42, yes, we will make ETag work. I'm not sure what the shape of that is but we will. |
Afterthought: I guess the second point could be remedied in conjunction with the |
@andig You can already strip prefixes when serving a filesystem over HTTP. I agree that the live-reloading can be done by a third party library wrapping an |
One more thing: if I understand correctly, embed will consider files locally to the package and forbids |
You can create a emed.go file in your assets directory and make the assets available as it's own package to the rest of your program. |
Agreed. In my opinion, adding syntactical sugar in the language to support this tooling change is a language change. I'm sure this is obvious to others, but this is effectively comment-as-code. I strongly feel that magic/sugar detracts from the simplicity and readability of the language; it is very easy to miss a magical comment that embeds a file. While a response to this could easily be "okay, then don't use it", this change means that a reviewer still has to be vigilant for others using this feature and has to remember that comments around variable declarations can break builds or fail at compile-time. I believe this is going to add confusion, detract from language usability, and will result in opaque, large binaries without clear benefit (regarding the lattermost concern, this will even lead to an anti-pattern of re-building binaries due to plain file changes). If |
@tristanfisher, I understand your point about language vs tooling change. It's certainly near the line. The reason that I consider it more a tooling change is that the language spec is unaffected - whether a program is valid does not change, the type checking process does not change. All that changes is the initial value of that variable following the comment. In this way it is a bit like the linker's -X flag, which can set the initial value of a top-level var of type string. It's fine for us to disagree; I just wanted to make my definition clear and explain the distinction I'm making. As for bloat, I guess we'll have to see, but I don't anticipate programs getting much larger than they already are. People already run tools that turn arbitrary files into Go code, check them into their repos, and make the compiler build them. The design removes some overhead from this process but does not enable anything new. Maybe people will abuse it now that it's easier to do, but on balance I don't expect that to be much of a problem. (And if some dependency embeds something so big that it bloats your binaries, you can always choose not to use that dependency.) As for rebuilds due to plain file changes, the only files that can trigger rebuilds are the ones in your own top-level module, since dependencies are immutable. If you found rebuilds happening more often than you'd like, the only explanation is (1) you are embedding files and (2) you are modifying those files. You would be in complete control of doing something about either cause. (It would be another thing entirely if a dependency's choice of what to use was somehow forcing extra rebuilds or other expense on you. But that's not the case here.) |
@rsc I agree that it's okay for us to disagree and I appreciate your response. My feeling is that if it's included by default in the standard tooling and comments can lead to an implicit initialization of a variable, then it's a language change. Outside of that debate, I guess my icky feeling is around more directives as "magic" comments that need to be memorized by (human) code readers. This could be taken to the absurd conclusion of adding new features via block comments that get handled at build time. That said, if this gets added to the ecosystem, I will be thankful that importing I think your point regarding the linker is a good one. It also helps explain my feelings on this: if the very end user (e.g. not someone that simply imports a package) is making the decision, there's no way to be surprised by blobs of non-code coming along for the ride. My concerns are born out of reviewing/pairing-on others' work and "tech-leady" responsibilities, which is why I felt the need to respond. I think "we'll have to see" sums it up well (I'm more cynical about bloat/misuse). |
I will read through the draft design tonight, so far it looks good from the TinyGo perspective. I just wanted to clarify one thing:
I don't know about U-root, but for TinyGo the main targets are microcontrollers which normally have far more flash than RAM (usually a factor of 8 or 16). A quick look at the draft design seems to suggest the idea is to keep the files in read-only memory, which would work fine for these targets: the embedded files can be read directly from flash. It would most likely not be desirable for TinyGo targets to decompress files at runtime. |
The io/fs proposal on which this depends looks to be blocked on Readdir/FileInfo problems, under discussion in #41188 and previously #40352. I've drafted an API to replace them in #41188 (comment) |
For #41191 Change-Id: I75d327759c3d9ef061c19a80b9b2619038dedf68 Reviewed-on: https://go-review.googlesource.com/c/go/+/281492 Trust: Ian Lance Taylor <iant@golang.org> Run-TryBot: Ian Lance Taylor <iant@golang.org> TryBot-Result: Go Bot <gobot@golang.org> Reviewed-by: Bryan C. Mills <bcmills@google.com>
Change https://golang.org/cl/285213 mentions this issue: |
For #40070 For #41191 For #43469 For #43632 Change-Id: I6dc6b6ea0f35876a4c252e4e287a0280aca9d502 Reviewed-on: https://go-review.googlesource.com/c/go/+/285213 Trust: Ian Lance Taylor <iant@golang.org> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
Apologies for not finding the discussion about why go source are not allowed to be go:embed. This prevents serving a directory of example go projects via go:embed, and keeps folks pinned to alternatives some of which aren't really maintained anymore.. Can you clarify if serving source files as text is a never vs a not yet, and if the former why? |
Go source files are allowed.
|
thanks, maybe the .go thing is was a red herring. It seems to be more about go.mod perhaps? Ex. Given a directory like this:
and an accessor file data/extension/init/templates.go
go build will do this:
if I change to
If I then delete the go.mod out of that directory, it works. So, I guess more concisely it is about go.mod, not files ending in .go. |
There's currently a glitch because go.mod isn't allowed in the go:embed path. Fixes #126 See golang/go#41191 (comment)
There's currently a glitch because go.mod isn't allowed in the go:embed path. Fixes #126 See golang/go#41191 (comment)
There's currently a glitch because go.mod isn't allowed in the go:embed path. Fixes #126 See golang/go#41191 (comment) Signed-off-by: Adrian Cole <adrian@tetrate.io>
It's a bit surprising, but looks like it's intended: go/src/cmd/go/internal/load/pkg.go Lines 2035 to 2036 in 87a3ac5
go/src/cmd/go/internal/load/pkg.go Lines 2053 to 2058 in 87a3ac5
|
This removes go.mod from the examples due to golang/go#41191 (comment) Signed-off-by: Adrian Cole <adrian@tetrate.io>
Matryoshka 😄 |
@warmchang correct and sorry to not summarize back. The issue is |
For future reference, to the next person starting to post here: This issue is closed. Comments might not get any audience. You are likely better served with alternatives:
|
In July, @bradfitz and I posted a draft design for embedded files. The doc links to a video, prototype code, and a Reddit discussion.
The feedback on that design has been overwhelmingly positive.
I propose to adopt the embedded files draft design for Go 1.16, with one addition, suggested in the discussion, to simplify the case of direct access to the bytes in a single embedded file.
As long as a file imports
"embed"
(import _ "embed"
if necessary), it will be permitted to use//go:embed
naming a single file (no glob patterns or directory matching allowed) to initialize a plainstring
or[]byte
variable:The import is required to flag the file as containing
//go:embed
lines and needing processing. Goimports (and gopls etc) can be taught this rule and automatically add the import in any file with a//go:embed
as needed.The embedded files design depends on the file system interface draft design, which I've also proposed to adopt in #41190.
This issue is only about adopting the embedded files design, under the assumption that the file system interface design is also adopted. If this proposal is accepted before the file system interface design is, we'd simply wait for the file system interface design before starting to land changes.
The text was updated successfully, but these errors were encountered: