subrequest proxy cache #125

Closed
biancalana opened this Issue Jun 9, 2012 · 7 comments

Comments

Projects
None yet
2 participants

Hi,

First, thank you for this awesome nginx module !

I'm writting some lua code to intercept all requests, dispatching then to a proxy cache internal location and make account of request rate, response status and cache status.

Everything is working except that cache isn't working.

Follow my nginx.conf:

worker_processes 1;

error_log /var/log/nginx-error.log debug;

events {
worker_connections 1024;
}

http {
include mime.types;

log_format full '$remote_addr $remote_user [$time_local] ' '"$host"->$proxy_host->$upstream_addr ' '"$request" $status($upstream_status) ' '$bytes_sent/$gzip_ratio($sent_http_content_type) ' '$request_time($upstream_response_time) ' '$upstream_cache_status';

access_log /var/log/nginx-access.log full;

sendfile on;
keepalive_timeout 65;

proxy_buffering on;
proxy_buffer_size 8k;
proxy_buffers 256 8k;
proxy_busy_buffers_size 64k;
proxy_temp_file_write_size 64k;
proxy_temp_path /opt/nginx/cache/tmp;
proxy_cache_path /opt/nginx/cache/ levels=1:2:2 keys_zone=STATIC:1000m inactive=1d max_size=4g;

location / {

default_type 'text/plain';

content_by_lua '

ngx.ctx.nuri = "/PROXY" .. ngx.var.request_uri;

local res = ngx.location.capture("/PROXY" .. ngx.var.request_uri,
                                                               { share_all_vars = true});

ngx.log(ngx.ERR, "cache_status(", ngx.var.upstream_cache_status, ")")

';
}

location /PROXY {
internal;

rewrite_by_lua 'local uri = ngx.re.sub(ngx.var.uri, "^/PROXY/(.*)", "/$1", "o")
ngx.req.set_uri(uri)
';

proxy_pass http://127.0.0.1:8080;
}

When a request is issued, the subrequest is done and the request hit the backend but ngx.var.upstream_cache_status variable is still empty... I can see that the response from the backend is begin stored in cache directory, but the subsequent requests are not served from cache and still hit the backend

I had never done anything like that and don't know if it's supposed to work, anybody has already did something like ?

Owner

agentzh commented Jun 11, 2012

Hello!

On Sat, Jun 9, 2012 at 11:58 PM, biancalana
reply@reply.github.com
wrote:

 First, thank you for this awesome nginx module !

Glad you like it :)

   local res = ngx.location.capture("/PROXY" .. ngx.var.request_uri,
                                                                  { share_all_vars = true});

   ngx.log(ngx.ERR, "cache_status(", ngx.var.upstream_cache_status, ")")
 ';

I'm afraid the $upstream_cache_status variable is a special variable
that is not just a value store. So you should read it in the context
of subrequest rather than the parent request, or the get_handler of
$upstream_cache_status (and its friends) will not get the right
request object. You can pass the value of $upstream_cache_status from
within the subrequest to the parent request by means of your custom
response headers (for example, you can use ngx_headers_more module to
insert response headers in the subrequest location, but you should
also be careful about the relative module loading order for
ngx_headers_more and ngx_lua).

Best regards,
-agentzh

Hi agentzh!

I tried to put:

    more_set_headers "X: $upstream_cache_status";

inside /PROXY (subrequest) location and fetch res.headers["X"] in parent location but it´s still does not work...

Owner

agentzh commented Jun 11, 2012

Hello!

On Mon, Jun 11, 2012 at 11:00 PM, biancalana
reply@reply.github.com
wrote:

 I tried to put:

       more_set_headers "X: $upstream_cache_status";

 inside /PROXY (subrequest) location and fetch res.headers["X"] in parent location but it´s still does not work...

As I said, you should be careful about module loading order, which
affects the running order of the output filters of ngx_lua and
ngx_headers_more. You should put
--add-module=/path/to/lua-ngingx-module before
--add-module=/path/to/headers-more-nginx-module while running
./configure for nginx, such that the output header filter of ngx_lua
will run after that of ngx_headers_more.

Regards,
-agentzh

Thank you for the suggestion, I had to recompile to meet this "order" but that still does not work.

Owner

agentzh commented Jun 12, 2012

Hello!

On Tue, Jun 12, 2012 at 9:27 AM, biancalana
reply@reply.github.com
wrote:

Thank you for the suggestion, I had to recompile to meet this "order" but that still does not work.

It seems that you didn't configure the "proxy_cache" directive in your
location /PROXY. Here is a complete working example that has been
tested on my side:

proxy_cache_path /tmp/cache levels=1 keys_zone=static:1m;

server {
    listen 8080;

    location /main {
        content_by_lua '
            local res = ngx.location.capture("/sub")
            ngx.say("cache-status: ", res.header["X-Cache-Status"])
        ';
    }

    location /sub {
        proxy_cache static;  # this line is crucial or the cache will be disabled!
        proxy_pass http://127.0.0.1:$server_port/index.html;
        more_set_headers "X-Cache-Status: $upstream_cache_status";
    }

    location /index.html {
        expires 10d;
    }
}

Then GET /main for the first time yields:

$ curl localhost:8080/main
cache-status: MISS

and subsequent accesses all give

$ curl localhost:8080/main
cache-status: HIT

Here's the nginx ./configure command line that I'm using:

./configure --add-module=/home/agentzh/git/lua-nginx-module \
                 --add-module=/home/agentzh/git//headers-more-nginx-module

Could you please try this sample on your side?

Best regards,
-agentzh

Owner

agentzh commented Aug 11, 2012

Any updates on this issue?

Owner

agentzh commented Oct 15, 2012

Consider it resolved.

agentzh closed this Oct 15, 2012

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment