body_filter_by_lua_block works with curl but not with browsers

[theMiddleBlue]

Hi,

I've a problem using body_filter_by_lua_block, it works just for requests by curl and not for browsers:

My config:

location / {
   proxy_pass http://wordpress:80;

   body_filter_by_lua_block {
      local from,to,err = ngx.re.find(ngx.var.request_filename, '\\/wp\\-includes\\/.*\\.php$')
      if from then
         local f,t,e = ngx.re.find(ngx.arg[1], '.*undefined.*')
         if f then
            ngx.arg[1] = 'PHP Error removed by policy\n'
            ngx.arg[2] = true
            return
         end
      end
   }
}

From curl:

From browser:

nginx -V:

nginx version: openresty/1.13.6.2
built by gcc 7.3.0 (Ubuntu 7.3.0-16ubuntu3) 
built with OpenSSL 1.1.0g  2 Nov 2017
TLS SNI support enabled
configure arguments: --prefix=/usr/local/openresty/nginx --with-cc-opt=-O2 --add-module=../ngx_devel_kit-0.3.0 --add-module=../echo-nginx-module-0.61 --add-module=../xss-nginx-module-0.06 --add-module=../ngx_coolkit-0.2rc3 --add-module=../set-misc-nginx-module-0.32 --add-module=../form-input-nginx-module-0.12 --add-module=../encrypted-session-nginx-module-0.08 --add-module=../srcache-nginx-module-0.31 --add-module=../ngx_lua-0.10.13 --add-module=../ngx_lua_upstream-0.07 --add-module=../headers-more-nginx-module-0.33 --add-module=../array-var-nginx-module-0.05 --add-module=../memc-nginx-module-0.19 --add-module=../redis2-nginx-module-0.15 --add-module=../redis-nginx-module-0.3.7 --add-module=../rds-json-nginx-module-0.15 --add-module=../rds-csv-nginx-module-0.09 --add-module=../ngx_stream_lua-0.0.5 --with-ld-opt=-Wl,-rpath,/usr/local/openresty/luajit/lib --add-module=/opt/openresty-1.13.6.2/../ModSecurity-nginx --with-stream --with-stream_ssl_module --with-http_ssl_module

Am I doing something wrong?

thanks

[theMiddleBlue]

uhm... it seems related to Accept-Encoding request header:

any idea?

[theMiddleBlue]

Ok, I've solved with a specific location and removing the Accept-Encoding header:

location ~ /wp\-includes/* {
  more_clear_input_headers "Accept-Encoding";

  proxy_pass http://wordpress:80;

  body_filter_by_lua_block {
    local from,to,err = ngx.re.find(ngx.var.request_filename, '\\/wp\\-includes\\/.*\\.php$')
    if from then
      local f,t,e = ngx.re.find(ngx.arg[1], '.*undefined.*')
      if f then
        ngx.arg[1] = 'PHP Error removed by policy\n'
        ngx.arg[2] = true
        return
      end
    end
  }
}

is there a better way to doing it?

[p0pr0ck5]

Likely what's going on is that Nginx is receiving a compressed response back from the upstream, and so you are calling ngx.re.find on compressed data, which is almost guaranteed to fail to match. If you want to filter the body like this than you will indeed to make sure the upstream does not send a compressed body (or you will need to buffer + decompress the body yourself, which seems silly), so the approach here isn't a bad idea IMO.

[tokers]

Basically the difference always comes from the different HTTP request headers. You may hide the Accept-Encoding to upstream by the directive:

proxy_set_header Accept-Encoding "";