Autoupdate osquery #34
Conversation
The upstream package was refactored and doesn't exist anymore. The functionality is actually being implemented in #34 so the correct dependencies will be added there. Because the updater package is missing, the dependency manager is corrupting the dependency cache for various pull requests. Closes #39
|
|
||
| // bootstraps local TUF metadata from bindata assets. | ||
| func (u *Updater) createLocalTufRepo() error { | ||
| if err := os.MkdirAll(u.settings.LocalRepoPath, 0755); err != nil { |
There was a problem hiding this comment.
This function doesn't take the creation of the directory structure for delegate roles into account.
├── root.json
├── snapshot.json
├── targets
│ ├── bar.json
│ ├── role
│ │ └── foo.json
│ └── role.json
├── targets.json
└── timestamp.json
There was a problem hiding this comment.
I'm going to move this to an issue
Makefile
Outdated
| INSECURE ?= false | ||
| PLATFORM ?= darwin | ||
| generate: | ||
| go run $(shell pwd)/autoupdate/generate_tuf.go \ |
There was a problem hiding this comment.
can't this be go run ./autoupdate/generate_tuf.go or even using GOPATH instead of pwd?
|
Should we consider merging this before too many merge conflicts arise? It looks like the debug logging changes maybe have caused some merge conflicts in |
|
No, I'm in the process of rebasing this locally. Started yesterday. |
autoupdate/autoupdate.go
Outdated
|
|
||
| func (u *Updater) binary() string { | ||
| bin := filepath.Base(string(u.destination)) | ||
| if bin == "osqueryd" { |
There was a problem hiding this comment.
@marpaia right now we have the binary named osqueryd but the download url uses osquery for the binary/url path.
Would you rather have this conditional in the code, or use osqueryd in the url on the mirror?
Let's 🚲 🏠
There was a problem hiding this comment.
I'm not sure that I follow. I think the binary should be called osqueryd in GCS.
cmd/launcher/launcher.go
Outdated
| @@ -59,6 +66,11 @@ func parseOptions() (*options, error) { | |||
| false, | |||
| "print launcher version and exit", | |||
| ) | |||
| flTLS = flag.Bool( | |||
There was a problem hiding this comment.
Let's rename this to something like flInsecureTLS as right now false indicates "use TLS" and that's a bit confusing.
Makefile
Outdated
|
|
||
|
|
||
| test: generate | ||
| go test -cover -v $(shell go list ./... | grep -v /vendor/) |
autoupdate/autoupdate.go
Outdated
| ) (*Updater, error) { | ||
| gun, err := d.gun() | ||
| if err != nil { | ||
| return nil, err |
There was a problem hiding this comment.
It would be nice to be consistent with the use of errors.Wrap (as in line 78). I'd prefer using it everywhere, but I'm open to argument.
autoupdate/autoupdate.go
Outdated
| localRepo := filepath.Base(u.settings.LocalRepoPath) | ||
| roles := []string{"root.json", "snapshot.json", "timestamp.json", "targets.json"} | ||
| for _, role := range roles { | ||
| asset, err := Asset(path.Join("autoupdate/assets", localRepo, role)) |
There was a problem hiding this comment.
split "autoupdate/assets" into two strings so path.Join can use the appropriate separator
autoupdate/autoupdate.go
Outdated
| func (u *Updater) Run(opts ...tuf.Option) (stop func(), err error) { | ||
| updaterOpts := []tuf.Option{ | ||
| tuf.WithHTTPClient(u.client), | ||
| tuf.WithFrequency(10 * time.Second), //TODO leave default |
There was a problem hiding this comment.
Is this a TODO for this PR or later?
There was a problem hiding this comment.
good question. I might just up this to every 5 minutes an issue for now while we're working out the details.
or i could expose it in a flag..
The default interval is 1 hour, which is ok for a customer's environment, but a hell of a time to wait while testing.
There was a problem hiding this comment.
When you refer to "testing" are you talking about manual testing, or automated tests? If it's manual testing then I think I agree with the idea of putting it behind a flag.
There was a problem hiding this comment.
Yes, manual testing with the launcher running and actually updating the artifact in the remote mirror.
I'll add the flag.
autoupdate/generate_tuf.go
Outdated
| flag.Parse() | ||
|
|
||
| gun := fmt.Sprintf("kolide/%s", *flBinary) | ||
| localRepo := filepath.Join("./autoupdate/assets/", fmt.Sprintf("%s-tuf", *flBinary)) |
There was a problem hiding this comment.
Can you clarify what you mean by split?
autoupdate/generate_tuf.go
Outdated
| ) | ||
| flag.Parse() | ||
|
|
||
| gun := fmt.Sprintf("kolide/%s", *flBinary) |
There was a problem hiding this comment.
Is this a path being created? Should it use filepath.Join?
There was a problem hiding this comment.
it's a URL path.
I suppose path.Join can work.
|
|
||
| // call this method to restart the launcher when autoupdate completes. | ||
| launcherFinalizer := func() error { | ||
| if err := syscall.Exec(os.Args[0], os.Args, os.Environ()); err != nil { |
cmd/launcher/launcher.go
Outdated
| logFatal(logger, errors.Wrap(err, "launching osquery instance")) | ||
| } | ||
|
|
||
| // TODO delete this block and use docker compose |
There was a problem hiding this comment.
yes, this was just a shim so I can test with a local mirror
Makefile
Outdated
| go run ./autoupdate/generate_tuf.go \ | ||
| -binary=osqueryd -notary=${NOTARY_URL} -insecure=${INSECURE} | ||
| # go run ./autoupdate/generate_tuf.go \ | ||
| # -binary=launcher -notary=${NOTARY_URL} -insecure=${INSECURE} |
There was a problem hiding this comment.
Should these be uncommented?
| updater.bootstrapFn = updater.createLocalTufRepo | ||
|
|
||
| for _, opt := range opts { | ||
| opt(&updater) |
cmd/launcher/launcher.go
Outdated
| launcherUpdaterOpts..., | ||
| ) | ||
| if err != nil { | ||
| logFatal(logger, "err", err) |
There was a problem hiding this comment.
I think that enableAutoUpdate should return (func(), err) and func main should be the one doing the logFataling.
Adds the autoupdate package which wraps github.com/kolide/updater/tuf to provide secure autoupdates from a remote mirror. If enabled, both osquery and launcher itself are updated when new versions are released.
This PR is mostly complete. Needs some work adding tests and using docker compose instead of some hacky code to run a local test mirror.