Versioning of build tools

[andrewkroh]

Our builds use a number of external tools that are fetched remotely. Things like go get and python virtualenv both do fetching. Some of these tools are unversioned (meaning that we don't specify a version to fetch). This can causes stability issues when the external tool changes.

And in an effort to further improve stability we might want to vendor the Go based tools directly in our repo or in S3 so that we are in full control over everything that goes into our build.

The first step here is to audit all of the places where we pull in remote code or tools so we can gauge the scope of the issue.

[andrewkroh]

Here are some quick notes about remote fetches. I didn't not look at any Dockerfiles in the project and we have lots of them that are used for testing.

• Packaging
  • Uses versioned Docker images from the Elastic registry. Source is at https://github.com/elastic/golang-crossbuild.
  • These images include
    • FPM 1.10.0
    • Go 1.10.3
    • Debian 8 and 9 base layers with several build tool packages pulled from official debian repos.
• Packetbeat
  • libpcap source from an Elastic S3 bucket
    • https://s3.amazonaws.com/beats-files/deps/libpcap-1.8.1.tar.gz
• All Beats
  • go get golang.org/x/tools/cmd/goimports
  • go get github.com/golang/lint/golint
  • go get github.com/haya14busa/reviewdog/cmd/reviewdog
  • go get github.com/elastic/go-licenser
  • go get github.com/mitchellh/gox
  • go get github.com/docker/libcompose

Python

$ find . -name requirements.txt | xargs cat | sort | uniq | grep -v '=='
elasticsearch
pylibmc
requests

[jsoriano]

Regarding go tools, mage is vendorized, and installed with go install from libbeat/scripts/Makefile. We could do the same with other go tools (if we do other thing then we should probably do the same with mage)

[jsoriano]

I think most of this is being solved by @kvch with the migration to go modules (see #16127, #15998 and so on).

[botelastic]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[elasticmachine]

Pinging @elastic/integrations (Team:Integrations)

[botelastic]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.