Install tools (mage, goimports, etc.) from vendor folder

[kvch]

What does this PR do?

This PR adds a new file tools/tools.go to list the tools we need during checking, building and packaging Beats. This is the suggested approach to list tool dependencies: https://github.com/golang/go/wiki/Modules#how-can-i-track-tool-dependencies-for-a-module

The following tools are added:

• github.com/magefile/mage
• github.com/pierrre/gotestcover
• github.com/tsg/go-daemon
• golang.org/x/tools/cmd/goimports
• github.com/elastic/go-licenser

Why is it important?

The tools are required to run make check on the CI.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made the corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works

[blakerouse]

Looks good to me, like the including of the tools.

Probably should let someone with a little more experience then me on the project give the +1. I am excited to see this completed, along with the python3 work as well.

[urso]

I see, we copied gotool from go-txfile to the Beats repo. Btw. go-txfile magefile uses the gotools package (minor improvements + generalization of this approach) from https://github.com/urso/magetools.

[urso]

why has this been removed?

[kvch]

It has been removed from master in the meantime: #15683

[urso]

I'm missing at least 'stringer' as developer dependency.

We use go:generate in a few places. Grepping code for go:generate I get this:

licenses/license.go
20://go:generate go run ../dev-tools/cmd/license/license_generate.go

metricbeat/module/windows/service/doc.go
21://go:generate go run ../../../helper/windows/run.go -cmd "go tool cgo -godefs defs_service_windows.go" -goarch amd64 -output defs_service_windows_amd64.go
22://go:generate go run ../../../helper/windows/run.go -cmd "go tool cgo -godefs defs_service_windows.go" -goarch 386 -output defs_service_windows_386.go
23://go:generate go run $GOROOT/src/syscall/mksyscall_windows.go -output zservice_windows.go service_windows.go
24://go:generate gofmt -w defs_service_windows_amd64.go defs_service_windows_386.go

metricbeat/helper/windows/pdh/doc.go
20://go:generate go run mkpdh_defs.go
21://go:generate go run ../run.go -cmd "go tool cgo -godefs defs_pdh_windows.go" -goarch amd64 -output defs_pdh_windows_amd64.go
22://go:generate go run ../run.go -cmd "go tool cgo -godefs defs_pdh_windows.go" -goarch 386 -output defs_pdh_windows_386.go
23://go:generate go run $GOROOT/src/syscall/mksyscall_windows.go -output zpdh_windows.go pdh_windows.go
24://go:generate gofmt -w defs_pdh_windows_amd64.go defs_pdh_windows_386.go zpdh_windows.go

filebeat/input/syslog/input.go
38://go:generate ragel -Z -G2 parser.rl -o parser.go
39://go:generate go fmt parser.go

auditbeat/module/file_integrity/flatbuffers.go
32://go:generate flatc --go schema.fbs

auditbeat/module/file_integrity/security_windows.go
70://go:generate go run $GOROOT/src/syscall/mksyscall_windows.go -output zsecurity_windows.go security_windows.go

winlogbeat/sys/eventlogging/syscall_windows.go
139://go:generate go run $GOROOT/src/syscall/mksyscall_windows.go -output zsyscall_windows.go syscall_windows.go

winlogbeat/sys/wineventlog/syscall_windows.go
303://go:generate go run $GOROOT/src/syscall/mksyscall_windows.go -output zsyscall_windows.go syscall_windows.go

packetbeat/procs/syscall_windows.go
72://go:generate go run $GOROOT/src/syscall/mksyscall_windows.go -output zsyscall_windows.go syscall_windows.go

libbeat/idxmgmt/idxmgmt.go
74://go:generate stringer -linecomment -type LoadMode

libbeat/idxmgmt/std.go
71://go:generate stringer -linecomment -type componentType

libbeat/feature/stability.go
20://go:generate stringer -type=Stability

libbeat/monitoring/monitoring.go
36://go:generate stringer -type=Mode

x-pack/filebeat/input/netflow/decoder/fields/doc.go
7://go:generate go run gen.go -output zfields_ipfix.go -export IpfixFields --column-id=1 --column-name=2 --column-type=3 ipfix-information-elements.csv
8://go:generate go run gen.go -output zfields_cert.go -export CertFields --column-pen=1 --column-id=2 --column-name=3 --column-type=4 cert_pen6871.csv
9://go:generate go run gen.go -output zfields_cisco.go -export CiscoFields --column-pen=2 --column-id=3 --column-name=1 --column-type=4 cisco.csv
10://go:generate go run gen.go -output zfields_assorted.go -export AssortedFields --column-pen=1 --column-id=2 --column-name=3 --column-type=4 assorted.csv
11://go:generate go fmt

x-pack/filebeat/input/netflow/doc.go
7://go:generate go run fields_gen.go -output _meta/fields.yml --column-name=2 --column-type=3 --header _meta/fields.header.yml decoder/fields/ipfix-information-elements.csv

x-pack/filebeat/module/cisco/shared/gen.go
16://go:generate go run gen-ftd-ecs-mapping.go stringset.go -output ecs-mapping-processor.yml security-mappings.csv
17://go:generate go run gen-ecs-mapping-docs.go stringset.go -output ecs-mapping-docs.asciidoc security-mappings.csv

x-pack/filebeat/processors/decode_cef/cef/cef.go
15://go:generate ragel -Z -G1 cef.rl -o parser.go
16://go:generate go fmt parser.go
20://go:generate ragel -V -p cef.rl -o cef.dot
21://go:generate dot -T svg cef.dot -o cef.svg

x-pack/filebeat/build/package/module/cisco/shared/gen.go
16://go:generate go run gen-ftd-ecs-mapping.go stringset.go -output ecs-mapping-processor.yml security-mappings.csv
17://go:generate go run gen-ecs-mapping-docs.go stringset.go -output ecs-mapping-docs.asciidoc security-mappings.csv

x-pack/libbeat/licenser/types.go
10://go:generate stringer -type=LicenseType -linecomment=true
24://go:generate stringer -type=State

x-pack/libbeat/management/api/response.go
13://go:generate stringer -type=LicenseType -linecomment=true

Other dependencies: stringer, ragel, dot, flatc, gofmt.

Looks like we inconsistently use gofmt and go fmt. How about replacing them with goimports?

Not all developer dependencies are go tools. We should document them somewhere. e.g. in the readme?

[kvch]

@urso I have replaced gofmt and go fmt with goimports.

I am wondering if we need to add dot, ragel and flatc needs to be versioned at all. Both dot and ragel are supposed to be installed using apt/homebrew/yum/whatever. Are we currently tracking the version of such non-Go tools? If not, do we want to?
flatc is supposed to be compiled if my research was correct. The questions above apply to this case as well.

[urso]

We should at least mention the versions that have been used somehwere. If generated files contain version info in the file header this might be good enough. These generated files are not updated often and newer version of developer tools might introduce breaking changes, that you only recognize after a long time. In this case it is nice to know which version was working last in case you need to fix something ASAP.

[kvch]

I am adding the required tools and versions to our contribution guide: https://www.elastic.co/guide/en/beats/devguide/current/beats-contributing.html

[kvch]

I have opened a PR against master with the documentation changes. Now I am movint the PR to "In review" again.