SetBuffer OnRequestBody return 400 Bad Request occasionally

[Xunzhuo]

When I try to change request body, I got 400 response occasionally. Sometimes envoy respond in expectation and sometimes respond in 400 Bad request

I use curl -d "Hello,World" localhost:18000/ to send request

Envoy log is below:

[2021-08-02 17:19:08.426][3314310][debug][conn_handler] [source/server/active_tcp_listener.cc:328] [C2] new connection
[2021-08-02 17:19:08.426][3314310][debug][http] [source/common/http/conn_manager_impl.cc:261] [C2] new stream
[2021-08-02 17:19:08.426][3314310][debug][http] [source/common/http/conn_manager_impl.cc:882] [C2][S7336547734171909877] request headers complete (end_stream=false):
':authority', 'localhost:18000'
':path', '/'
':method', 'POST'
'user-agent', 'curl/7.61.1'
'accept', '*/*'
'content-length', '11'
'content-type', 'application/x-www-form-urlencoded'

[2021-08-02 17:19:08.426][3314310][debug][router] [source/common/router/router.cc:445] [C2][S7336547734171909877] cluster 'httpbin' match for URL '/'
[2021-08-02 17:19:08.426][3314310][debug][router] [source/common/router/router.cc:631] [C2][S7336547734171909877] router decoding headers:
':authority', 'localhost:18000'
':path', '/'
':method', 'POST'
':scheme', 'http'
'user-agent', 'curl/7.61.1'
'accept', '*/*'
'content-length', '11'
'content-type', 'application/x-www-form-urlencoded'
'x-forwarded-proto', 'http'
'x-request-id', 'c89942ff-89da-461e-a70d-bed81b3efbe6'
'x-envoy-expected-rq-timeout-ms', '15000'

[2021-08-02 17:19:08.426][3314310][debug][pool] [source/common/conn_pool/conn_pool_base.cc:232] [C1] using existing connection
[2021-08-02 17:19:08.426][3314310][debug][pool] [source/common/conn_pool/conn_pool_base.cc:171] [C1] creating stream
[2021-08-02 17:19:08.426][3314310][debug][router] [source/common/router/upstream_request.cc:371] [C2][S7336547734171909877] pool ready
[2021-08-02 17:19:08.426][3314310][debug][http] [source/common/http/filter_manager.cc:779] [C2][S7336547734171909877] request end stream
[2021-08-02 17:19:08.426][3314310][debug][router] [source/common/router/router.cc:1244] [C2][S7336547734171909877] upstream headers complete: end_stream=false
[2021-08-02 17:19:08.426][3314310][debug][http] [source/common/http/conn_manager_impl.cc:1469] [C2][S7336547734171909877] encoding headers via codec (end_stream=false):
':status', '400'
'content-type', 'text/plain; charset=utf-8'
'x-envoy-upstream-service-time', '0'
'date', 'Mon, 02 Aug 2021 09:19:08 GMT'
'server', 'envoy'

[2021-08-02 17:19:08.426][3314310][debug][connection] [source/common/network/connection_impl.cc:633] [C1] remote close
[2021-08-02 17:19:08.426][3314310][debug][connection] [source/common/network/connection_impl.cc:243] [C1] closing socket: 0
[2021-08-02 17:19:08.426][3314310][debug][client] [source/common/http/codec_client.cc:124] [C1] response complete
[2021-08-02 17:19:08.426][3314310][debug][pool] [source/common/http/http1/conn_pool.cc:53] [C1] response complete
[2021-08-02 17:19:08.426][3314310][debug][pool] [source/common/http/http1/conn_pool.cc:59] [C1] saw upstream close connection
[2021-08-02 17:19:08.426][3314310][debug][client] [source/common/http/codec_client.cc:101] [C1] disconnect. resetting 0 pending requests
[2021-08-02 17:19:08.426][3314310][debug][pool] [source/common/conn_pool/conn_pool_base.cc:393] [C1] client disconnected, failure reason: 
[2021-08-02 17:19:08.426][3314310][debug][pool] [source/common/conn_pool/conn_pool_base.cc:200] [C1] destroying stream: 1 remaining
[2021-08-02 17:19:08.426][3314310][debug][connection] [source/common/network/connection_impl.cc:633] [C2] remote close
[2021-08-02 17:19:08.426][3314310][debug][connection] [source/common/network/connection_impl.cc:243] [C2] closing socket: 0
[2021-08-02 17:19:08.426][3314310][debug][conn_handler] [source/server/active_tcp_listener.cc:76] [C2] adding to cleanup list

wasm c++ code:

#include <string>
#include <string_view>
#include <unordered_map>

#include "proxy_wasm_intrinsics.h"

class ExampleContext : public Context {
public:
  explicit ExampleContext(uint32_t id, RootContext *root) : Context(id, root) {}
  FilterDataStatus onRequestBody(size_t body_buffer_length, bool end_of_stream) override;

};
static RegisterContextFactory register_ExampleContext(CONTEXT_FACTORY(ExampleContext),
                                                      "my_root_id");

FilterDataStatus ExampleContext::onRequestBody(size_t body_buffer_length,
                                               bool /* end_of_stream */) {
  setBuffer(WasmBufferType::HttpRequestBody, 0, body_buffer_length , "Hello Envoy!");
  return FilterDataStatus::Continue;
}

envoy config is:

static_resources:
  listeners:
    - name: main
      address:
        socket_address:
          address: 0.0.0.0
          port_value: 18000
      filter_chains:
        - filters:
            - name: envoy.http_connection_manager
              typed_config:
                "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
                stat_prefix: ingress_http
                codec_type: auto
                route_config:
                  name: httpbin
                  virtual_hosts:
                    - name: httpbin
                      domains:
                        - "*"
                      routes:
                        - match:
                            prefix: "/"
                          route:
                            cluster: httpbin
                http_filters:
                  - name: envoy.filters.http.wasm
                    typed_config:
                      "@type": type.googleapis.com/envoy.extensions.filters.http.wasm.v3.Wasm
                      config:
                        name: "my_plugin"
                        root_id: "my_root_id"
                        vm_config:
                          vm_id: "my_vm_id"
                          runtime: "envoy.wasm.runtime.v8"
                          code:
                            local:
                              filename: "./resbody.wasm"
                          allow_precompiled: true
              
                  - name: envoy.filters.http.router
                    typed_config: {}

  clusters:
    - name: httpbin
      connect_timeout: 5000s
      type: strict_dns
      lb_policy: round_robin
      load_assignment:
        cluster_name: httpbin
        endpoints:
          - lb_endpoints:
              - endpoint:
                  address:
                    socket_address:
                      address: 0.0.0.0
                      port_value: 9000

admin:
  access_log_path: "/dev/null"
  address:
    socket_address:
      address: 0.0.0.0
      port_value: 8002

upstream is coded by Golang http package:

package main

import (
	"fmt"
	"io/ioutil"
	"log"
	"net/http"
)

func hello(w http.ResponseWriter, req *http.Request) {
	requestMessage(w, req)
	log.Println("Get connection Success!")
	w.Write([]byte("Hello Envoy at 0.0.0.0:9000!"))
}

func requestMessage(w http.ResponseWriter, req *http.Request) {
	for name, headers := range req.Header {
		for _, h := range headers {
			fmt.Fprintf(w, "%v: %v\n", name, h)
		}
	}
	s, err := ioutil.ReadAll(req.Body)
	if err != nil {
		fmt.Fprintf(w, "Read Body failed")
		return
	}
	fmt.Fprintf(w, "Request body is: %s\n", s)
}

func main() {
	log.Println("Start to listen 0.0.0.0:9000, prefix: /")
	http.HandleFunc("/", hello)
	http.ListenAndServe(":9000", nil)
}

Expected response from upstream is:

Sometimes it return:

[Xunzhuo]

@PiotrSikora @mathetake This problem confused me, please check this when u are free.

[PiotrSikora]

@Xunzhuo you're replacing request body, possibly mutliple times (OnRequestBody is called for each request body chunk), but you don't modify the headers that include Content-Length header, so the updated body doesn't match expected length, hence 400 Bad Request.

If you want to replace complete body, you need to remove Content-Length header (or replace it if you know the correct value ahead of time), and then you can either wait until end_of_stream is true, or simply replace request body the first callback, and discard future chunks in subsequent calls, but you need to code that in your plugin, since that's not something that works out of the box right now.

[Xunzhuo]

@PiotrSikora Thanks, problems have been solved by your guide😊

[coolg92003]

@PiotrSikora Thanks, problems have been solved by your guide😊

How do u change the Content-Length header in onRequestBody(), according to WASM doc, it can't modify HTTP header in onRequestBody().
Would you share your code?
thanks
Cliff