POC wasm as upstream filter #33119

juanmolle · 2024-03-26T15:51:18Z

When the filter is trying to be created. it try to get the clusterManager reference from ServerFactoryContext but it seems it is not available at that moment the pointer is null. not initialized at that moment.

[2024-03-26 12:35:53.520][144854083][critical][assert] [source/server/server.cc:144] assert failure: config_.clusterManager() != nullptr.
[2024-03-26 12:35:53.520][144854083][critical][backtrace] [./source/server/backtrace.h:104] Caught Abort trap: 6, suspect faulting address 0x182652a60
[2024-03-26 12:35:53.520][144854083][critical][backtrace] [./source/server/backtrace.h:91] Backtrace (use tools/stack_decode.py to get line numbers):
[2024-03-26 12:35:53.520][144854083][critical][backtrace] [./source/server/backtrace.h:92] Envoy version: 22e6a78/1.30.0-dev/Modified/DEBUG/BoringSSL
[2024-03-26 12:35:53.541][144854083][critical][backtrace] [./source/server/backtrace.h:96] #0: Envoy::SignalAction::sigHandler() [0x10c6dc9b8]
[2024-03-26 12:35:53.541][144854083][critical][backtrace] [./source/server/backtrace.h:96] #1: _sigtramp [0x1826bb584]
[2024-03-26 12:35:53.541][144854083][critical][backtrace] [./source/server/backtrace.h:96] #2: pthread_kill [0x18268ac20]
[2024-03-26 12:35:53.541][144854083][critical][backtrace] [./source/server/backtrace.h:96] #3: abort [0x182597a20]
[2024-03-26 12:35:53.545][144854083][critical][backtrace] [./source/server/backtrace.h:96] #4: Envoy::Server::InstanceBase::clusterManager() [0x10a3f5bcc]
[2024-03-26 12:35:53.547][144854083][critical][backtrace] [./source/server/backtrace.h:96] #5: Envoy::Server::ServerFactoryContextImpl::clusterManager() [0x10a374710]
[2024-03-26 12:35:53.548][144854083][critical][backtrace] [./source/server/backtrace.h:96] #6: Envoy::Extensions::HttpFilters::Wasm::FilterConfig::FilterConfig()

To make it work I just mock the clusterManager and could create a policy to add a header as upstream filter.
I have to add a check to not make the clearCacheRoute, I guess this has no meaning as upstream. Probably other checks are needed in the code.
Due to mock the clusterManager, I understand some functionality is not available like making calls to any server. not sure if has sens in an upstream filter.

I don't know how we could solve the clusterManager issue.

More Context in the stack

[2024-03-26 12:35:53.509][144854083][debug][misc] [source/common/network/dns_resolver/dns_factory_util.cc:39] create Apple DNS resolver type: envoy.network.dns_resolver.apple in MacOS.
[2024-03-26 12:35:53.509][144854083][debug][misc] [source/common/network/dns_resolver/dns_factory_util.cc:75] create DNS resolver type: envoy.network.dns_resolver.apple
[2024-03-26 12:35:53.519][144854083][debug][config] [./source/common/http/filter_chain_helper.h:111] upstream http filter #0
[2024-03-26 12:35:53.520][144854083][critical][assert] [source/server/server.cc:144] assert failure: config_.clusterManager() != nullptr.
[2024-03-26 12:35:53.520][144854083][critical][backtrace] [./source/server/backtrace.h:104] Caught Abort trap: 6, suspect faulting address 0x182652a60
[2024-03-26 12:35:53.520][144854083][critical][backtrace] [./source/server/backtrace.h:91] Backtrace (use tools/stack_decode.py to get line numbers):
[2024-03-26 12:35:53.520][144854083][critical][backtrace] [./source/server/backtrace.h:92] Envoy version: 22e6a78/1.30.0-dev/Modified/DEBUG/BoringSSL
[2024-03-26 12:35:53.541][144854083][critical][backtrace] [./source/server/backtrace.h:96] #0: Envoy::SignalAction::sigHandler() [0x10c6dc9b8]
[2024-03-26 12:35:53.541][144854083][critical][backtrace] [./source/server/backtrace.h:96] #1: _sigtramp [0x1826bb584]
[2024-03-26 12:35:53.541][144854083][critical][backtrace] [./source/server/backtrace.h:96] #2: pthread_kill [0x18268ac20]
[2024-03-26 12:35:53.541][144854083][critical][backtrace] [./source/server/backtrace.h:96] #3: abort [0x182597a20]
[2024-03-26 12:35:53.545][144854083][critical][backtrace] [./source/server/backtrace.h:96] #4: Envoy::Server::InstanceBase::clusterManager() [0x10a3f5bcc]
[2024-03-26 12:35:53.547][144854083][critical][backtrace] [./source/server/backtrace.h:96] #5: Envoy::Server::ServerFactoryContextImpl::clusterManager() [0x10a374710]
[2024-03-26 12:35:53.548][144854083][critical][backtrace] [./source/server/backtrace.h:96] #6: Envoy::Extensions::HttpFilters::Wasm::FilterConfig::FilterConfig() [0x10553fe64]
[2024-03-26 12:35:53.550][144854083][critical][backtrace] [./source/server/backtrace.h:96] #7: Envoy::Extensions::HttpFilters::Wasm::FilterConfig::FilterConfig() [0x1055403d8]
[2024-03-26 12:35:53.551][144854083][critical][backtrace] [./source/server/backtrace.h:96] #8: std::__1::construct_at[abi:v160006]<>() [0x10553ac3c]
[2024-03-26 12:35:53.553][144854083][critical][backtrace] [./source/server/backtrace.h:96] #9: std::__1::allocator_traits<>::construct[abi:v160006]<>() [0x10553aa54]
[2024-03-26 12:35:53.555][144854083][critical][backtrace] [./source/server/backtrace.h:96] #10: std::__1::__shared_ptr_emplace<>::__shared_ptr_emplace[abi:v160006]<>() [0x10553a978]
[2024-03-26 12:35:53.557][144854083][critical][backtrace] [./source/server/backtrace.h:96] #11: std::__1::__shared_ptr_emplace<>::__shared_ptr_emplace[abi:v160006]<>() [0x10553a658]
[2024-03-26 12:35:53.558][144854083][critical][backtrace] [./source/server/backtrace.h:96] #12: std::__1::allocate_shared[abi:v160006]<>() [0x10553a558]
[2024-03-26 12:35:53.560][144854083][critical][backtrace] [./source/server/backtrace.h:96] #13: std::__1::make_shared[abi:v160006]<>() [0x1055397b8]
[2024-03-26 12:35:53.561][144854083][critical][backtrace] [./source/server/backtrace.h:96] #14: Envoy::Extensions::HttpFilters::Wasm::WasmFilterConfig::createFilterFactoryFromProtoTyped() [0x105539718]
[2024-03-26 12:35:53.563][144854083][critical][backtrace] [./source/server/backtrace.h:96] #15: Envoy::Extensions::HttpFilters::Common::DualFactoryBase<>::createFilterFactoryFromProto() [0x105539cdc]
[2024-03-26 12:35:53.565][144854083][critical][backtrace] [./source/server/backtrace.h:96] #16: Envoy::Http::FilterChainHelper<>::processFilter() [0x10b0bed64]
[2024-03-26 12:35:53.567][144854083][critical][backtrace] [./source/server/backtrace.h:96] #17: Envoy::Http::FilterChainHelper<>::processFilters() [0x10b054b38]
[2024-03-26 12:35:53.569][144854083][critical][backtrace] [./source/server/backtrace.h:96] #18: Envoy::Upstream::ClusterInfoImpl::ClusterInfoImpl() [0x10b0507a8]
[2024-03-26 12:35:53.571][144854083][critical][backtrace] [./source/server/backtrace.h:96] #19: Envoy::Upstream::ClusterInfoImpl::ClusterInfoImpl() [0x10b0550b8]
[2024-03-26 12:35:53.572][144854083][critical][backtrace] [./source/server/backtrace.h:96] #20: Envoy::Upstream::ClusterImplBase::ClusterImplBase() [0x10b056550]
[2024-03-26 12:35:53.574][144854083][critical][backtrace] [./source/server/backtrace.h:96] #21: Envoy::Upstream::BaseDynamicClusterImpl::?::ClusterImplBase() [0x1044ba19c]
[2024-03-26 12:35:53.576][144854083][critical][backtrace] [./source/server/backtrace.h:96] #22: Envoy::Upstream::StrictDnsClusterImpl::StrictDnsClusterImpl() [0x1045393cc]
[2024-03-26 12:35:53.577][144854083][critical][backtrace] [./source/server/backtrace.h:96] #23: Envoy::Upstream::StrictDnsClusterImpl::StrictDnsClusterImpl() [0x104539fac]
[2024-03-26 12:35:53.579][144854083][critical][backtrace] [./source/server/backtrace.h:96] #24: std::__1::construct_at[abi:v160006]<>() [0x104547a70]
[2024-03-26 12:35:53.580][144854083][critical][backtrace] [./source/server/backtrace.h:96] #25: std::__1::allocator_traits<>::construct[abi:v160006]<>() [0x10454785c]
[2024-03-26 12:35:53.582][144854083][critical][backtrace] [./source/server/backtrace.h:96] #26: std::__1::__shared_ptr_emplace<>::__shared_ptr_emplace[abi:v160006]<>() [0x104547778]
[2024-03-26 12:35:53.583][144854083][critical][backtrace] [./source/server/backtrace.h:96] #27: std::__1::__shared_ptr_emplace<>::__shared_ptr_emplace[abi:v160006]<>() [0x104547450]
[2024-03-26 12:35:53.585][144854083][critical][backtrace] [./source/server/backtrace.h:96] #28: std::__1::allocate_shared[abi:v160006]<>() [0x104547348]
[2024-03-26 12:35:53.586][144854083][critical][backtrace] [./source/server/backtrace.h:96] #29: std::__1::make_shared[abi:v160006]<>() [0x10453ace0]
[2024-03-26 12:35:53.588][144854083][critical][backtrace] [./source/server/backtrace.h:96] #30: Envoy::Upstream::StrictDnsClusterFactory::createClusterImpl() [0x10453abb4]
[2024-03-26 12:35:53.590][144854083][critical][backtrace] [./source/server/backtrace.h:96] #31: Envoy::Upstream::ClusterFactoryImplBase::create() [0x10b0f53d0]
[2024-03-26 12:35:53.591][144854083][critical][backtrace] [./source/server/backtrace.h:96] #32: Envoy::Upstream::ClusterFactoryImplBase::create() [0x10b0f4a38]
[2024-03-26 12:35:53.593][144854083][critical][backtrace] [./source/server/backtrace.h:96] #33: Envoy::Upstream::ProdClusterManagerFactory::clusterFromProto() [0x10a5a6e50]
[2024-03-26 12:35:53.595][144854083][critical][backtrace] [./source/server/backtrace.h:96] #34: Envoy::Upstream::ClusterManagerImpl::loadCluster() [0x10a58c7c0]
[2024-03-26 12:35:53.597][144854083][critical][backtrace] [./source/server/backtrace.h:96] #35: Envoy::Upstream::ClusterManagerImpl::init() [0x10a58a9ec]
[2024-03-26 12:35:53.599][144854083][critical][backtrace] [./source/server/backtrace.h:96] #36: Envoy::Upstream::ProdClusterManagerFactory::clusterManagerFromProto() [0x10a5a3a3c]
[2024-03-26 12:35:53.600][144854083][critical][backtrace] [./source/server/backtrace.h:96] #37: Envoy::Server::Configuration::MainImpl::initialize() [0x10bc37e9c]
[2024-03-26 12:35:53.602][144854083][critical][backtrace] [./source/server/backtrace.h:96] #38: Envoy::Server::InstanceBase::initializeOrThrow() [0x10a3fdae0]
[2024-03-26 12:35:53.604][144854083][critical][backtrace] [./source/server/backtrace.h:96] #39: Envoy::Server::InstanceBase::initialize() [0x10a3f886c]
[2024-03-26 12:35:53.606][144854083][critical][backtrace] [./source/server/backtrace.h:96] #40: Envoy::createFunction()::$_1::operator()() [0x10a3011a0]

Signed-off-by: Juan Manuel Ollé <jolle@mulesoft.com>

repokitteh-read-only · 2024-03-26T15:51:23Z

As a reminder, PRs marked as draft will not be automatically assigned reviewers,
or be handled by maintainer-oncall triage.

Please mark your PR as ready when you want it to be reviewed!

🐱

Caused by: #33119 was opened by juanmolle.

see: more, trace.

juanmolle · 2024-03-26T15:53:29Z

source/extensions/filters/http/wasm/wasm_filter.cc

@@ -23,7 +149,7 @@ FilterConfig::FilterConfig(const envoy::extensions::filters::http::wasm::v3::Was
    });
  };

-  if (!Common::Wasm::createWasm(plugin, context.scope().createScope(""), server.clusterManager(),
+  if (!Common::Wasm::createWasm(plugin, context.scope().createScope(""), mycluster /*server.clusterManager()*/,


here is the main issue I found. server.clusterManager() assert due to the manager is still not initialized (I guess) at this moment

Where is this assert failing? was it a manual run of Envoy with upstream filter? If it's in a unit test, the unit test is probably just not set up correctly. if you add an e2e test with wasm upstream filters it will give me something Ican patch and run.

Usually I suggest folks doing upstream filter work parameterize the e2e test using prependFilter like we did for test/integration/filter_integration_test.cc

@alyssawilk Just creating the filter in normal envoy work, it tried to get the cluster manager from the server factory and it is not initialized. I guess @yanjunxiang-google get the same issue while working to convert composite filter and propose a change to solve it #33221

Yeah, that's the same issue: #33218, which is resolved now.

juanmolle · 2024-03-26T15:54:07Z

test/extensions/common/aws/credentials_provider_impl_test.cc

@@ -183,6 +183,30 @@ TEST_F(CredentialsFileCredentialsProviderTest, DefaultCredentialsFile) {
  EXPECT_EQ("profile1_token", credentials.sessionToken().value());
 }

+TEST_F(CredentialsFileCredentialsProviderTest, CustomProfileFromConfigShouldBeHonored) {


Ignore this, belong to other Patch i was working on

alyssawilk · 2024-03-28T13:54:41Z

source/extensions/filters/http/wasm/wasm_filter.cc

+  virtual Singleton::Manager& singletonManager() override { throw 1; }
+};
+
+struct MyClusterManager : public Upstream::ClusterManager {


is this used?

alyssawilk · 2024-03-28T14:11:22Z

source/extensions/filters/http/wasm/wasm_filter.cc

@@ -23,7 +149,7 @@ FilterConfig::FilterConfig(const envoy::extensions::filters::http::wasm::v3::Was
    });
  };

-  if (!Common::Wasm::createWasm(plugin, context.scope().createScope(""), server.clusterManager(),
+  if (!Common::Wasm::createWasm(plugin, context.scope().createScope(""), mycluster /*server.clusterManager()*/,


Where is this assert failing? was it a manual run of Envoy with upstream filter? If it's in a unit test, the unit test is probably just not set up correctly. if you add an e2e test with wasm upstream filters it will give me something Ican patch and run.

Usually I suggest folks doing upstream filter work parameterize the e2e test using prependFilter like we did for test/integration/filter_integration_test.cc

alyssawilk · 2024-04-03T13:05:44Z

ah cool, let's wait on that then :-)

juanmolle · 2024-04-04T16:28:58Z

This is the issue to handle cluster_manager initialization order

#33218

POC wasm as upstream filter

357df68

Signed-off-by: Juan Manuel Ollé <jolle@mulesoft.com>

juanmolle mentioned this pull request Mar 26, 2024

Convert wasm to dual filter #33055

Closed

juanmolle commented Mar 26, 2024

View reviewed changes

alyssawilk reviewed Mar 28, 2024

View reviewed changes

juanmolle closed this Apr 12, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

POC wasm as upstream filter #33119

POC wasm as upstream filter #33119

juanmolle commented Mar 26, 2024

repokitteh-read-only bot commented Mar 26, 2024

juanmolle Mar 26, 2024

alyssawilk Mar 28, 2024

juanmolle Apr 3, 2024

yanjunxiang-google Apr 15, 2024

juanmolle Mar 26, 2024

alyssawilk Mar 28, 2024

alyssawilk Mar 28, 2024

alyssawilk commented Apr 3, 2024

juanmolle commented Apr 4, 2024 •

edited

Loading

POC wasm as upstream filter #33119

POC wasm as upstream filter #33119

Conversation

juanmolle commented Mar 26, 2024

repokitteh-read-only bot commented Mar 26, 2024

juanmolle Mar 26, 2024

Choose a reason for hiding this comment

alyssawilk Mar 28, 2024

Choose a reason for hiding this comment

juanmolle Apr 3, 2024

Choose a reason for hiding this comment

yanjunxiang-google Apr 15, 2024

Choose a reason for hiding this comment

juanmolle Mar 26, 2024

Choose a reason for hiding this comment

alyssawilk Mar 28, 2024

Choose a reason for hiding this comment

alyssawilk Mar 28, 2024

Choose a reason for hiding this comment

alyssawilk commented Apr 3, 2024

juanmolle commented Apr 4, 2024 • edited Loading

juanmolle commented Apr 4, 2024 •

edited

Loading