Skip to content

golang-leipzig/file-embedding

master
Switch branches/tags

Name already in use

A tag already exists with the provided branch name. Many Git commands accept both tag and branch names, so creating this branch may cause unexpected behavior. Are you sure you want to create this branch?
Code

Latest commit

 

Git stats

Files

Permalink
Failed to load latest commit information.
Type
Name
Latest commit message
Commit time
 
 
 
 
 
 
 
 

File Embedding in Go

Talk at Golang Leipzig November Meetup

What is File Embedding?

  • packing arbitrary resources into a binary/executable
  • a resource can be anything, e.g. HTML templates, CSS or Javascript files, a favicon, translations...
  • example layout of a typical web application:
├── assets
│   ├── base.css
│   ├── base.js
│   ├── ...
│   └── favicon.svg
├── migrations
│   ├── 01_initial_setup.down.sql
│   └── ...
├── notes.go
├── storage.go
├── storage_test.go
└── views
    ├── index.gohtml
    ├── ...
    └── layouts
        └── base.gohtml

Advantages 👍

  • only a single file needs to be distributed/deployed
  • especially useful for desktop applications (no need for $XDG_HOME, %APPDATA% or ~/Library)
  • embedded data is "immutable" (at least from outside the process)

Disadvantages 👎

  • binary size grows with each resource that gets embedded → increased memory usage
  • space efficiency depends on the encoding of the embedded content
  • large files are not suitable for embedding

If the resources are compressable---e.g. most plain text files are---then a binary packer like upx can reduce the file size dramatically:

$ upx -o notes.upx notes
$ du -sh notes notes.upx 
24M     notes
9.7M    notes.upx

Available Implementations

  • go-bindata most popular but now deprecated, recommends pkger
  • pkger
    • requires a Go modules project
    • API simulates os.File
    • 👎 stores a lot of redundant metadata, e.g. absolute paths to files 👮‍♀️ or imported packages
  • statik
    • simulates an http.FileSystem---ok for some use cases but pretty otherwise inconvenient
    • 👎 can only include a single directory
  • go.rice
    • 👎 complicated to use, scans your source code for rice.FindBox("/path") calls and includes them
    • 👎 weird limitations, e.g. paths cannot be constant strings or undefined behaviour when called in init()
    • failed to set it up for my example application, I really wonder why this got popular 🤷‍♂️

It's Time for Yet Another File Embedding Tool

Design

Usage

NAME:
   embed - A new cli application

USAGE:
   embed [global options] command [command options] [arguments...]

VERSION:
   unset

COMMANDS:
   help, h  Shows a list of commands or help for one command

GLOBAL OPTIONS:
   --package value, -p value                    name of the package the generated Go file is associated to (default: "main")
   --destination value, --dest value, -d value  where to store the generated Go file (default: "embeds.go")
   --include value, -i value                    paths to embed, directories are stored recursively (can be used multiple times)
   --help, -h                                   show help (default: false)
   --version, -v                                print the version (default: false)

Draft for Embedded Static Files

The draft---published by Russ Cox and Brad Fitzpatrick in July---identified the following problems with the current situation:

  • too many existing tools 😅
  • all depend on a manual generation step
  • generated file bloats git history (with a second slightly larger copy of each file)
  • not go-gettable if generated file is not checked into source control

Adding support for file embedding to the go command will eliminate those problems.

Proposed changes:

  • new //go:embed directive
  • embed package containing embed.Files that implements fs.FS file system interface draft which makes it directly usable with net/http and hmtl/template

Here's an example for the embed directive:

// content holds our static web server content.
//go:embed image/* template/*
//go:embed html/index.html
var content embed.Files

The draft also contains various considerations regarding:

Still curious?

About

A talk about the what, why and how of file embedding in Go.

Topics

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published