Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Separate management and functions in Functionbeat #12939

Merged
merged 4 commits into from Aug 5, 2019
Merged

Separate management and functions in Functionbeat #12939

merged 4 commits into from Aug 5, 2019

Conversation

kvch
Copy link
Contributor

@kvch kvch commented Jul 17, 2019 •
edited

This PR splits Functionbeat into multiple parts and prepares it for supporting multiple cloud providers.

The existing binary is split into two parts: manager and AWS cloud function. The responsibility of the manager is to deploy, update and delete the cloud function. The cloud function obviously the binary run by the provider.

The packages were rearranged so no unnecessary files are compiled into cloud functions. See dependencies below:

image

(Note: GCP is not yet added to Functionbeat.)

  • config: configuration of Functionbeat
  • manager: common management code and provider specific code under provider
  • function: common function code
  • provider: provider specific code for cloud functions by provider

TODO

  • test cloud function

@kvch kvch force-pushed the feature-functionbeat-separate-by-provider branch from 994d161 to 335a6c8 Compare July 17, 2019 10:42
@kvch kvch marked this pull request as ready for review July 17, 2019 15:30
@kvch kvch requested a review from a team as a code owner July 17, 2019 15:30
@ph ph self-requested a review July 17, 2019 15:31
@kvch kvch force-pushed the feature-functionbeat-separate-by-provider branch from 6d1e798 to d2d1f0d Compare July 18, 2019 09:14
@kvch
Copy link
Contributor Author

kvch commented Jul 19, 2019

Failing tests are unrelated.

ph
ph reviewed Jul 23, 2019
View reviewed changes
Copy link
Contributor

@ph ph left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kvch I like what you did with the code it make it more flexible and allow us to effectively develop for GCP and remove unnecessary cruft from the binaries. I just added a few simple comment, nothing really actionable on your part. I wanted to test the actual binary with the newly generated packages but I can't seems to build them locally.

I've tried the mage package in the x-pack/functionbeat/ directory which works in master and produce binaries that work, but this doesn't seem to work in the current form of this pull request. Maybe I am missing something on my side?

>> package: Building functionbeat type=zip for platform=windows/386
>> package: Building functionbeat type=tar.gz for platform=linux/amd64
package ran for 51.627773588s
Error: failed building functionbeat type=zip for platform=windows/amd64: failed adding file={Source:provider/aws/build/golang-crossbuild/functionbeat-aws-linux-amd64 Content: Template: Target:pkg/functionbeat-aws Mode:-rwxr-xr-x Config:false Modules:false Dep:<nil> Owner:} to zip: lstat provider/aws/build/golang-crossbuild/functionbeat-aws-linux-amd64: no such file or directory
failed building functionbeat type=tar.gz for platform=linux/amd64: failed adding file={Source:provider/aws/build/golang-crossbuild/functionbeat-aws-linux-amd64 Content: Template: Target:pkg/functionbeat-aws Mode:-rwxr-xr-x Config:false Modules:false Dep:<nil> Owner:} to tar: lstat provider/aws/build/golang-crossbuild/functionbeat-aws-linux-amd64: no such file or directory
failed building functionbeat type=tar.gz for platform=linux/386: failed adding file={Source:provider/aws/build/golang-crossbuild/functionbeat-aws-linux-amd64 Content: Template: Target:pkg/functionbeat-aws Mode:-rwxr-xr-x Config:false Modules:false Dep:<nil> Owner:} to tar: lstat provider/aws/build/golang-crossbuild/functionbeat-aws-linux-amd64: no such file or directory
failed building functionbeat type=zip for platform=windows/386: failed adding file={Source:provider/aws/build/golang-crossbuild/functionbeat-aws-linux-amd64 Content: Template: Target:pkg/functionbeat-aws Mode:-rwxr-xr-x Config:false Modules:false Dep:<nil> Owner:} to zip: lstat provider/aws/build/golang-crossbuild/functionbeat-aws-linux-amd64: no such file or directory
failed building functionbeat type=tar.gz for platform=darwin/amd64: failed adding file={Source:provider/aws/build/golang-crossbuild/functionbeat-aws-linux-amd64 Content: Template: Target:pkg/functionbeat-aws Mode:-rwxr-xr-x Config:false Modules:false Dep:<nil> Owner:} to tar: lstat provider/aws/build/golang-crossbuild/functionbeat-aws-linux-amd64: no such file or directory

x-pack/functionbeat/manager/aws/event_stack_poller.go Show resolved Hide resolved
x-pack/functionbeat/manager/cmd/cli_handler_test.go Show resolved Hide resolved
x-pack/functionbeat/manager/cmd/provider_cmd.go Outdated Show resolved Hide resolved
x-pack/functionbeat/manager/cmd/provider_cmd.go Outdated Show resolved Hide resolved
x-pack/functionbeat/manager/executor/executor.go Show resolved Hide resolved
x-pack/functionbeat/provider/local/main_test.go Outdated Show resolved Hide resolved
@kvch
Copy link
Contributor Author

kvch commented Jul 26, 2019 •
edited

I cannot reproduce the packaging issue you are facing. Could you please share the output of mage -v package? Also, the contents of provider/aws/build/golang-crossbuild? I assume something goes wrong, because you are running darwin.

I am able to reproduce the issue.

@ph
Copy link
Contributor

ph commented Jul 26, 2019

@kvch

I have tested the build all the artifact are build correctly

winterfell~/go/src/github.com/elastic/beats/x-pack/functionbeat/build/distributions(feature-functionbeat-separate-by-provider|…) % tar -xzvf functionbeat-8.0.0-linux-x86_64.tar.gz
functionbeat-8.0.0-linux-x86_64/functionbeat
functionbeat-8.0.0-linux-x86_64/LICENSE.txt
functionbeat-8.0.0-linux-x86_64/pkg/functionbeat-aws
functionbeat-8.0.0-linux-x86_64/fields.yml
functionbeat-8.0.0-linux-x86_64/functionbeat.yml
functionbeat-8.0.0-linux-x86_64/README.md
functionbeat-8.0.0-linux-x86_64/NOTICE.txt
functionbeat-8.0.0-linux-x86_64/functionbeat.reference.yml
functionbeat-8.0.0-linux-x86_64/.build_hash.txt

I have tested the deploy binary with --help and it was executed.
I have some issues with my AWS account so I cannot test the deploy.

@kvch
Copy link
Contributor Author

kvch commented Jul 29, 2019

Failing tests are unrelated.

@kvch
Copy link
Contributor Author

kvch commented Jul 29, 2019

I tested it, and after the small fix i've pushed, is working.

@kvch
Copy link
Contributor Author

kvch commented Aug 5, 2019

Failing tests are unrelated.

@kvch kvch merged commit 23cd13a into elastic:master Aug 5, 2019
jsoriano
jsoriano reviewed Aug 7, 2019
View reviewed changes
x-pack/metricbeat/magefile.go
var Aliases = map[string]interface{}{
"goTestUnit": GoUnitTest, // dev-tools/jenkins_ci.ps1 uses this.
}

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kvch was this code removed by mistake? Windows builds are failing with Unknown target specified: goTestUnit. I have opened PR to recover it #13190

@kvch kvch mentioned this pull request Aug 28, 2019
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jsoriano jsoriano jsoriano left review comments

@houndci-bot houndci-bot houndci-bot left review comments

@ph ph ph approved these changes

Assignees
No one assigned
Labels
Functionbeat review
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
4 participants
@kvch @ph @jsoriano @houndci-bot