Vendor full aws-sdk instead of individual packages

[andrewvc]

This takes over from #12548 , and resolves a lot of packaging weirdness by simply importing the full AWS SDK with govendor. This also updates the version to v0.9.0 from preview.5. This is a prereq for #12401

govendor doesn't play well when attempting to update the individual packages, complaining frequently about subtree overwrites. With this change, an update is as simple as govendor fetch github.com/aws/aws-sdk-go-v2/...@VERSION.

Previously I used a bash one liner to parse the package names out of vendor.json and attempt to invoke govendor once per package. However, that hit the aforementioned problems with subtree overwrites.

I've heard that there are some concerns about this potentially increasing binary size. I've tried compiling metricbeat from on this branch. The binary was 56 megabytes. Identical to master.

[elasticmachine]

Pinging @elastic/uptime

[kaiyan-sheng]

How did you measure the new binary metricbeat size? When I tried mage update; mage build under x-pack/metricbeat with v0.9.0 aws-sdk-go-v2, it failed building the binary becasue:

>> build: Building metricbeat
# github.com/elastic/beats/x-pack/metricbeat/module/aws
module/aws/aws.go:102:21: undefined: ec2iface.EC2API
module/aws/utils.go:29:78: undefined: cloudwatchiface.CloudWatchAPI
module/aws/utils.go:46:100: undefined: cloudwatchiface.CloudWatchAPI
module/aws/utils.go:63:79: undefined: cloudwatchiface.CloudWatchAPI
Error: running "go build -o metricbeat -ldflags -X github.com/elastic/beats/libbeat/version.buildTime=2019-06-19T16:39:55Z -X github.com/elastic/beats/libbeat/version.commit=19c14dc51fe0ba8023394b61d65475291b1ed157" failed with exit code 2

So if we decide to go with v0.9.0, then I need to fix aws module first 😬

[andrewvc]

@kaiyan-sheng weird! I just used make metricbeat... hmmmmmmmmmmm let me try again with mage

[andrewvc]

@kaiyan-sheng works for me

➜  metricbeat git:(unify-aws-packages) rm metricbeat
➜  metricbeat git:(unify-aws-packages) stat metricbeat
stat: cannot stat 'metricbeat': No such file or directory
➜  metricbeat git:(unify-aws-packages) mage build
>> build: Building metricbeat
➜  metricbeat git:(unify-aws-packages) stat metricbeat
  File: metricbeat
  Size: 58535408  	Blocks: 114328     IO Block: 4096   regular file
Device: 1000005h/16777221d	Inode: 8664408212  Links: 1
Access: (0755/-rwxr-xr-x)  Uid: (  501/andrewcholakian)   Gid: (   20/   staff)
Access: 2019-06-19 12:29:02.673278323 -0500
Modify: 2019-06-19 12:29:01.829555131 -0500
Change: 2019-06-19 12:29:01.830320695 -0500
 Birth: 2019-06-19 12:29:01.280176443 -0500

[andrewvc]

Functionbeat is broken I should say though, I'll try and fix that

[kaiyan-sheng]

Did you run the command under beats/x-pack/metricbeat?

[andrewvc]

Ah, I see the issue was that I was using the OSS version.

[exekias]

This is great! this change will require to update Functionbeat and Metricbeat AWS module, do you plan to do that?

As a side note, it would probably make a lot of sense to unify how we initialize AWS client, so we all share the same parameters set, like we did with Kubernetes or Docker: https://github.com/elastic/beats/blob/master/libbeat/common/docker/client.go

[andrewvc]

@exekias Glad to hear it! I just pushed the updates to functionbeat/metricbeat to make them compatible. I also merged in the latest master.

I agree with unifying the initialization, but perhaps we can do that in a separate PR? The scope here is large as is.

We also will need a separate PR to take advantage of the newer version of the AWS library's support for adding a context.Context to most ops. For now I just stubbed in context.TODO(). This doesn't regress the functionality, but we should add configurable timeouts in a future PR.

[exekias]

That sounds great to me, thank you for taking the load here!

[andrewvc]

@kaiyan-sheng @exekias I'm thinking I'm going to put this PR on indefinite hold and use preview5 for my ELB PR. I got pretty far here, but now I'm getting weird NPEs in functionbeat. I don't have the time to sink into this right now. Maybe @ph can take a look?

[kaiyan-sheng]

@andrewvc Thanks for the update! I can also take a look later for the unit tests in metricbeat aws module that are still failing with v0.9.0, for example https://travis-ci.org/elastic/beats/jobs/550513776#L647

[exekias]

cc @kvch FYI

[kvch]

I have tracked down the issue with Functionbeat tests. In aws.Request HTTPRequest cannot be null, so the context can be set.

 .../provider/aws/event_stack_poller_test.go         |  4 +++-
 .../provider/aws/op_cloudformation_test.go          | 21 +++++++++++++--------
 2 files changed, 16 insertions(+), 9 deletions(-)

diff --git a/x-pack/functionbeat/provider/aws/event_stack_poller_test.go b/x-pack/functionbeat/provider/aws/event_stack_poller_test.go
index 6f72d4737..ab1ee3725 100644
--- a/x-pack/functionbeat/provider/aws/event_stack_poller_test.go
+++ b/x-pack/functionbeat/provider/aws/event_stack_poller_test.go
@@ -5,6 +5,7 @@
 package aws

 import (
+   "net/http"
    "strconv"
    "testing"
    "time"
@@ -52,8 +53,9 @@ func (m *mockCloudFormationClient) DescribeStackEventsRequest(
            m.Index++
        }
    }()
+   httpReq, _ := http.NewRequest("", "", nil)
    return cloudformation.DescribeStackEventsRequest{
-       Request: &aws.Request{Data: &m.Responses[m.Index]},
+       Request: &aws.Request{Data: &m.Responses[m.Index], HTTPRequest: httpReq},
    }
 }

diff --git a/x-pack/functionbeat/provider/aws/op_cloudformation_test.go b/x-pack/functionbeat/provider/aws/op_cloudformation_test.go
index ddc0aec4d..04c57df68 100644
--- a/x-pack/functionbeat/provider/aws/op_cloudformation_test.go
+++ b/x-pack/functionbeat/provider/aws/op_cloudformation_test.go
@@ -6,6 +6,7 @@ package aws

 import (
    "errors"
+   "net/http"
    "testing"

    "github.com/aws/aws-sdk-go-v2/aws"
@@ -40,14 +41,15 @@ func (m *mockCloudformationStack) CreateStackRequest(
        m.onCreateStackInput(input)
    }

+   httpReq, _ := http.NewRequest("", "", nil)
    if m.err != nil {
        return cloudformation.CreateStackRequest{
-           Request: &aws.Request{Data: m.respCreateStackOutput, Error: m.err},
+           Request: &aws.Request{Data: m.respCreateStackOutput, Error: m.err, HTTPRequest: httpReq},
        }
    }

    return cloudformation.CreateStackRequest{
-       Request: &aws.Request{Data: m.respCreateStackOutput},
+       Request: &aws.Request{Data: m.respCreateStackOutput, HTTPRequest: httpReq},
    }
 }

@@ -58,14 +60,15 @@ func (m *mockCloudformationStack) DeleteStackRequest(
        m.onDeleteStackInput(input)
    }

+   httpReq, _ := http.NewRequest("", "", nil)
    if m.err != nil {
        return cloudformation.DeleteStackRequest{
-           Request: &aws.Request{Data: m.respDeleteStackOutput, Error: m.err},
+           Request: &aws.Request{Data: m.respDeleteStackOutput, Error: m.err, HTTPRequest: httpReq},
        }
    }

    return cloudformation.DeleteStackRequest{
-       Request: &aws.Request{Data: m.respDeleteStackOutput},
+       Request: &aws.Request{Data: m.respDeleteStackOutput, HTTPRequest: httpReq},
    }
 }

@@ -76,14 +79,15 @@ func (m *mockCloudformationStack) DescribeStacksRequest(
        m.onDescribeStacksInput(input)
    }

+   httpReq, _ := http.NewRequest("", "", nil)
    if m.err != nil {
        return cloudformation.DescribeStacksRequest{
-           Request: &aws.Request{Data: m.respDescribeStacksOutput, Error: m.err},
+           Request: &aws.Request{Data: m.respDescribeStacksOutput, Error: m.err, HTTPRequest: httpReq},
        }
    }

    return cloudformation.DescribeStacksRequest{
-       Request: &aws.Request{Data: m.respDescribeStacksOutput},
+       Request: &aws.Request{Data: m.respDescribeStacksOutput, HTTPRequest: httpReq},
    }
 }

 @@ -94,14 +98,15 @@ func (m *mockCloudformationStack) UpdateStackRequest(
         m.onUpdateStackInput(input)
     }

 +   httpReq, _ := http.NewRequest("", "", nil)
     if m.err != nil {
         return cloudformation.UpdateStackRequest{
 -           Request: &aws.Request{Data: m.respUpdateStackOutput, Error: m.err},
 +           Request: &aws.Request{Data: m.respUpdateStackOutput, Error: m.err, HTTPRequest: httpReq},
         }
     }

     return cloudformation.UpdateStackRequest{
 -       Request: &aws.Request{Data: m.respUpdateStackOutput},
 +       Request: &aws.Request{Data: m.respUpdateStackOutput, HTTPRequest: httpReq},
     }
  }

I've mailed the patch to you.

[andrewvc]

Incorporated @kvch 's patch (thanks!) as well as merged in master and updated the new RDS stuff. 🤞

[kaiyan-sheng]

Sorry for the delay! I will fix the rest of the failing unit tests here!

[andrewvc]

jenkins, retest this please

[kaiyan-sheng]

@andrewvc TestGetListMetricsOutput is still failing, I will push a patch to fix that. Sorry!
For the filebeat test failure, if you rebase on top of the master branch, then it should pass with #12819 merged.

[kaiyan-sheng]

andrewvc#2 should fix the unit test.

[andrewvc]

jenkins, please retest this.

[andrewvc]

jenkins, retest this please.

[kaiyan-sheng]

andrewvc#3 one more please!

[kaiyan-sheng]

Yay the tests passed!