New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature request: allow manually setting Mod Time #66

Open
veqryn opened this Issue Feb 9, 2019 · 6 comments

Comments

Projects
None yet
2 participants
@veqryn
Copy link

veqryn commented Feb 9, 2019

Thank you for this package.

I currently use it to host a swagger.json file that contains the API documentation for my gRPC server (which has a REST->gRPC Gateway on it, hence the need for swagger).

We re-generate code from all of our protobuf files whenever they change, and whenever any mocked interfaces change (we just have a container that runs go generate against everything in the repo).

Because of this, the timestamps on our files (including the swagger.json files) are constantly changing, even though the contents of the files does not change.

This means that our vfsgen embedded doc files always show changes to the modTime values, even though nothing else changed.

I would like to disable this, and the easiest way would probably be to create an option to manually set the modTime to a specific value. I'm open to other idea too though.

thanks!

@dmitshur

This comment has been minimized.

Copy link
Member

dmitshur commented Feb 10, 2019

Hi @veqryn,

Thanks for the feature request. Fortunately, this is already possible to achieve without changes to vfsgen. vfsgen was designed to enable this kind of customization (and more) through interfaces.

Please see #31 (comment) and the code snippet inside. It should let you achieve what you want.

Edit: Since you want to set modtime to zero rather than modify it from zero, the Stat method would instead look more like this:

...

func (f modTimeFile) Stat() (os.FileInfo, error) {
    fi, err := f.File.Stat()
    if err != nil {
        return nil, err
    }
    return modTimeFileInfo{fi}, nil
}

type modTimeFileInfo struct {
    os.FileInfo
}

func (modTimeFileInfo) ModTime() time.Time {
    return time.Time{}
    // or any custom logic you'd like to implement for your needs
    // (perhaps use the modtime of the protobuf files you're generating from)
}

This is a common feature request, and so I plan to document it in the README so it's more visible. Issue #31 tracks that task.

@dmitshur dmitshur added the question label Feb 10, 2019

@veqryn

This comment has been minimized.

Copy link
Author

veqryn commented Feb 11, 2019

Hi @dmitshur, I'm just a little lost, but how does this work to prevent the generated file from having a different mod timestamp?

To clarify, I want the generated file to be the same each time I generate it (even if it has been modified/saved multiple times between), that way git stops telling me there are changes when the content has stayed the same.

@dmitshur

This comment has been minimized.

Copy link
Member

dmitshur commented Feb 12, 2019

Can you share how you're using vfsgen right now? If so, it'll be easier for me to show you what I mean. If not, I can try to explain anyway by coming up with an example.

@veqryn

This comment has been minimized.

Copy link
Author

veqryn commented Feb 12, 2019

Sure.

  1. I run vfsgen to generate embedded golang files.
  2. Commit the generated embedded golang files to our repo.
  3. Sometime later, I or someone else will recompile all of our various proto files. We have a lot, so we just recompile all of them, regardless of whether they have changed or not. Protoc is deterministic and has stable output, so if the contents of a proto file have not changed, then the outputted generated golang file from protoc will show no differences (ie: git diff shows nothing).
  4. Re-run vfsgen to re-create the embedded golang files.
  5. Go to commit to repo, and notice that vfs shows lots of timestamps have changed, even though the contents have not changed at all (ie: git diff shows lots of changed files). The timestamps have changed because protoc output identical but new files when it ran.

I would like it so that vfsgen had an option to overwrite the modtime when it generates files, in the actual files it generates, that way git doesn't show differences all the time when i go to commit those files.

(This has nothing to do with an http file server, as I don't use that at all in any step above.)

@veqryn

This comment has been minimized.

Copy link
Author

veqryn commented Feb 12, 2019

My code looks like this:

Golang executable to generate embedded golang files (because I couldn't understand what https://github.com/shurcooL/vfsgen/blob/master/cmd/vfsgendev/main.go was doing or how to use it):

package main

import (
	"flag"
	"fmt"
	"net/http"

	"github.com/shurcooL/vfsgen"
)

func main() {

	dir := flag.String("dir", "", "The directory to recursively generate vfs / embedded-bindata for")
	outfile := flag.String("outfile", "", "The file path and name (include extension) to output the generated file")
	pkg := flag.String("pkg", "", "The package name to give the vfs file")
	tags := flag.String("tags", "", "The build tags to give the vfs generation")
	variable := flag.String("variable", "", "The variable name to give the vfs (start with a capital letter if you want it exported)")
	comment := flag.String("comment", "", "The comment to give the variable")

	flag.Parse()

	fmt.Printf("vfsgen for directory: %s; output to: %s; package name: %s; build tags: %s; variable name: %s; comment: %s\n", *dir, *outfile, *pkg, *tags, *variable, *comment)
	err := vfsgen.Generate(http.Dir(*dir), vfsgen.Options{

		// Filename of the generated Go code output (including extension)
		Filename: *outfile,

		// PackageName is the name of the package in the generated code
		PackageName: *pkg,

		// BuildTags are optional build tags to give the generated code
		BuildTags: *tags,

		// VariableName is the name of the http.FileSystem variable in the generated code
		VariableName: *variable,

		// VariableComment is the comment of the http.FileSystem variable in the generated code
		VariableComment: *comment,
	})

	if err != nil {
		panic(err)
	}
}

An example of my go generate statements:

// Generate golang protobuf/grpc/gateway and swagger docs
//go:generate protoc -I=./include -I=../vendor/github.com/grpc-ecosystem/grpc-gateway/third_party/googleapis -I=../vendor/github.com/lyft -I=../vendor/github.com/grpc-ecosystem/grpc-gateway -I=./ecm/v2 --go_out=plugins=grpc:./ecm/v2/go --grpc-gateway_out=logtostderr=true:./ecm/v2/go --swagger_out=logtostderr=true:./ecm/v2/swagger --validate_out=lang=go:./ecm/v2/go ./ecm/v2/rp_ecm_v2.proto

// Generate embedded docs
// TODO: get vfsgen to ignore or skip if only the file timestamp has changed OR somehow overwrite the timestamp
//go:generate go run ./vfsgen/vfsgen.go --dir=./ecm/v2/swagger/ --outfile=./ecm/v2/embedded_docs/embedded_docs.go --pkg=embedded_docs --variable=Docs -comment "Docs statically implements an embedded virtual filesystem provided to vfsgen. To access the swagger file, use path: '/rp_ecm_v2.swagger.json'"

And then some code that is using the particular embedded golang file:

	// Open the embedded swagger json
	f, err := embedded_docs.Docs.Open("/rp_ecm_v2.swagger.json")
	// .... stuff ...
@dmitshur

This comment has been minimized.

Copy link
Member

dmitshur commented Feb 13, 2019

Thanks.

The relevant part in the snippet is here:

vfsgen.Generate(http.Dir(*dir), ...

That means you're currently using http.Dir as the http.FileSystem interface implementation that you provide to vfsgen.Generate.

Your original request was to be able to override the underlying filesystem (on disk) file mod times to be all zeros. So, you just need to wrap http.Dir(*dir) with the http.FileSystem wrapper I showed above. The wrapper will override file mod times to be zero, which is what vfsgen.Generate will then use.

It can look something like this:

package main

import (...)

func main() {
    ...
    // Override all file mod times to be zero using modTimeFS.
    var inputFS http.FileSystem = modTimeFS{
        fs: http.Dir(*dir),
    }
    err := vfsgen.Generate(inputFS, vfsgen.Options{
    ...
}

// modTimeFS is an http.FileSystem wrapper that modifies
// underlying fs such that all of its file mod times are set to zero.
type modTimeFS struct {
    fs http.FileSystem
}

func (fs modTimeFS) Open(name string) (http.File, error) {
    f, err := fs.fs.Open(name)
    if err != nil {
        return nil, err
    }
    return modTimeFile{f}, nil
}

type modTimeFile struct {
    http.File
}

func (f modTimeFile) Stat() (os.FileInfo, error) {
    fi, err := f.File.Stat()
    if err != nil {
        return nil, err
    }
    return modTimeFileInfo{fi}, nil
}

type modTimeFileInfo struct {
    os.FileInfo
}

func (modTimeFileInfo) ModTime() time.Time {
    return time.Time{}
    // or any custom logic you'd like to implement for your needs
    // (perhaps use the modtime of the protobuf files you're generating from)
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment