Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Upstream envoyproxy/envoy-wasm. #12546

Merged
merged 2 commits into from
Oct 9, 2020
Merged

Conversation

jplevyak
Copy link
Contributor

@jplevyak jplevyak commented Aug 7, 2020

For an explanation of how to fill out the fields, please see the relevant section
in PULL_REQUESTS.md

Commit Message: Upstream envoyproxy/envoy-wasm.
Additional Description: This PR represents the proxy-wasm work which has been going on over the last 18 months in the envoyproxy/envoy-wasm repository. The vast majority of the changes herein are confined to extensions which are optional. Changes outside of the extensions are the minimal set required in order to support those extensions.
Risk Level: Low (if the extensions are not included).
Testing: Unit tests in envoyproxy/envoy and integration tests in istio/proxy.
Docs Changes: Docs will be provided in subsequent PRs.
Release Notes: Wasm plugins available optionally.
[Optional Runtime guard:]
[Optional Fixes #Issue] #4272
[Optional Deprecated:]

@repokitteh-read-only
Copy link

CC @envoyproxy/api-shepherds: Your approval is needed for changes made to api/envoy/.
CC @envoyproxy/api-watchers: FYI only for changes made to api/envoy/.

🐱

Caused by: #12546 was opened by jplevyak.

see: more, trace.

@lizan lizan self-assigned this Aug 7, 2020
api/envoy/extensions/wasm/v3/wasm.proto Outdated Show resolved Hide resolved
api/envoy/extensions/wasm/v3/wasm.proto Show resolved Hide resolved
api/wasm/README.md Outdated Show resolved Hide resolved
bazel/BUILD Show resolved Hide resolved
bazel/repositories.bzl Outdated Show resolved Hide resolved
@jplevyak
Copy link
Contributor Author

The Linux-x64 release seems to be a flake.

I don't know that this code has ever been tested on Windows or arm64. Any suggestions on how to debug these things locally and/or disable the Wasm extensions for these environments?

.bazelrc Outdated Show resolved Hide resolved
@lizan
Copy link
Member

lizan commented Aug 10, 2020

@jplevyak we can skip arm / windows, just modify build_setup / do_ci script to add bazel flags.

@htuch
Copy link
Member

htuch commented Aug 10, 2020

@jplevyak how far does this get us to Wasm upstreaming? Is this mission complete? Would be good to document the support status after this PR in the commit message and version history.

@@ -0,0 +1,42 @@
Envoy hooks:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The api/ tree is for xDS APIs, why is there Wasm docs and build files here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is all going to be removed. Pending envoyproxy/envoy-wasm#613 which I will merge in here and ping this thread.

bazel/foreign_cc/llvm.patch Show resolved Hide resolved
"ASMFLAGS": "-UDEBUG",
},
lib_source = "@org_llvm_llvm//:all",
static_libraries = select({
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jplevyak
Copy link
Contributor Author

@htuch with this we will be 95% complete. There are a couple things which are still under development by interns and there is one thing that istio needs, but for all intents and purposes this will be enough for most users.

@jplevyak
Copy link
Contributor Author

@jplevyak we can skip arm / windows, just modify build_setup / do_ci script to add bazel flags.

Which flags do you mean? I don't think we have a flag to remove Wasm support i.e. we have --define wasm=enabled and --define wsam=wavm, but I don't think we have --define wasm=none.

@PiotrSikora
Copy link
Contributor

@jplevyak we can skip arm / windows, just modify build_setup / do_ci script to add bazel flags.

Which flags do you mean? I don't think we have a flag to remove Wasm support i.e. we have --define wasm=enabled and --define wsam=wavm, but I don't think we have --define wasm=none.

We used to have wasm=disabled before we enabled V8 by default. It should be in archives if you want to dig it up.

Copy link
Contributor

@ggreenway ggreenway left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please toggle the default value of whether wasm is compiled here, so it is tested in CI:

COMPILE_TIME_OPTIONS="\

tools/spelling/spelling_dictionary.txt Outdated Show resolved Hide resolved
source/server/admin/prometheus_stats.cc Outdated Show resolved Hide resolved
@ggreenway
Copy link
Contributor

@jplevyak we can skip arm / windows, just modify build_setup / do_ci script to add bazel flags.

Which flags do you mean? I don't think we have a flag to remove Wasm support i.e. we have --define wasm=enabled and --define wsam=wavm, but I don't think we have --define wasm=none.

Please add a flag for wasm=none. It's a huge dependency, and should be optional.

@jplevyak
Copy link
Contributor Author

@lizan I have made v8 support optional. We now need to add --define wasm=enabled to make it enabled. Which C builds should it default to being enabled?

@ggreenway
Copy link
Contributor

@lizan I have made v8 support optional. We now need to add --define wasm=enabled to make it enabled. Which C builds should it default to being enabled?

In general, most/all things default to enabled/included. Also, please the settings here: https://github.com/envoyproxy/envoy/blob/master/bazel/README.md#disabling-optional-features

"src/**/*.cc",
],
exclude = ["src/**/wavm*"],
# Note that the select cannot appear in the glob.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wavm has a cmake build system, why can't we build it under rules_foreign_cc rather than this handcrafted blob thing?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

WAVM is built using rules_foreign_cc, this here is a BUILD file for Proxy-Wasm C++ host implementation, which optionally includes WAVM support.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there any possibility here of not checking in this long glob file?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you want all the files called out separately for each case? It will be longer, but perhaps that is OK?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like you folks are contributors to https://github.com/proxy-wasm/proxy-wasm-cpp-host. Can you add some Bazel BUILD files there?

@PiotrSikora
Copy link
Contributor

PiotrSikora commented Aug 12, 2020

In general, most/all things default to enabled/included. Also, please the settings here: https://github.com/envoyproxy/envoy/blob/master/bazel/README.md#disabling-optional-features

So what's the default that we want to implement? V8 enabled by default, WAVM (only?) enabled with --define wasm=wavm, no Wasm with --define wasm=disabled? Do we also want both enabled with --define wasm=enabled?

@ggreenway
Copy link
Contributor

In general, most/all things default to enabled/included. Also, please the settings here: https://github.com/envoyproxy/envoy/blob/master/bazel/README.md#disabling-optional-features

So what's the default that we want to implement? V8 enabled by default, WAVM (only?) enabled with --define wasm=wavm, no Wasm with --define wasm=disabled? Do we also want both enabled with --define wasm=enabled?

I don't know the tradeoffs or considerations for V8 vs WAVM. Someone else can chime on that.

My request is just that there be a way to not compile either one in (which it looks like is now provided), and that we try to test as many variations as possible in CI.

@azure-pipelines
Copy link

Azure Pipelines successfully started running 1 pipeline(s).

@jplevyak
Copy link
Contributor Author

Only issue now is the MacOS CI

@mattklein123
Copy link
Member

OSX is messed up right now. We can force merge assuming this is ready to go and passes otherwise. Are there any remaining issues? cc @lizan

@mattklein123
Copy link
Member

@lizan should be back from vacation and will get this merged. Assigning over to him.

@PiotrSikora
Copy link
Contributor

PiotrSikora commented Oct 9, 2020

@mattklein123 @htuch all tests pass now, and this PR is synced with the current master, so it should be good to go.

@mattklein123
Copy link
Member

@PiotrSikora DCO is now broken. Can you just squash it all into a single commit and force push?

@mattklein123 mattklein123 self-assigned this Oct 9, 2020
@mattklein123
Copy link
Member

Note that we are going to merge this without docs but we need to do a doc pass before we do any major PR about this feature.
Ref: #13426 #9465 #9462

cc @louiscryan

Signed-off-by: John Plevyak <jplevyak@gmail.com>
Signed-off-by: Piotr Sikora <piotrsikora@google.com>
CODEOWNERS Outdated Show resolved Hide resolved
Signed-off-by: Piotr Sikora <piotrsikora@google.com>
@repokitteh-read-only repokitteh-read-only bot removed the api label Oct 9, 2020
@mattklein123 mattklein123 merged commit 319a9a6 into envoyproxy:master Oct 9, 2020
@duderino
Copy link

duderino commented Oct 9, 2020

huzzah!

@gbrail
Copy link
Contributor

gbrail commented Oct 12, 2020

Cool!

istio-testing pushed a commit to istio/proxy that referenced this pull request Oct 16, 2020
* use envoy master instead of envoy-wasm

envoy-wasm has been merged to envoy
envoyproxy/envoy#12546

* use istio/envoy release-1.8

as requested by @bianpengyuan
#3055 (comment)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet