limit readahead for private objects

[HLeithner]

Expected Behavior

varnish doesn't need additional memory

Current Behavior

consumes memory till oom

Possible Solution

pipe without caching?

Steps to Reproduce (for bugs)

1. create a big file (ex. 5gb) on the backend server
2. use a varnish with limited memory 2gb
3. use the following vcl

sub vcl_backend_response {
  // don't cache files larger than 20MB
  if (std.integer(beresp.http.Content-Length, 0) > 20242880) {
    set beresp.uncacheable = true;
    set beresp.do_gzip     = false;
    set beresp.do_gunzip   = false;
    set beresp.ttl         = 120s;
    return (deliver);
  }
}

4. varnish will run oom

Context

I'm trying to get around caching big files, in varnish < 4.0 I found code snipes for restarting and piping. But this doesn't work any longer in varnish >= 4.0. As alternative I tried 2 things.

In my opinion if I use uncacheable = true, varnish should not allocate memory for the data transfered.

Your Environment

• Version used: 6.2
• Operating System and version: Debian Stretch
• Source of binary packages used: official packages
• haproxy with UDS

[nigoroll]

I have not (yet) checked all the details, but this has to do with the fact that at some point we changed how uncacheable fetches work: While, in some very old code, we looped over a fixed size (relatively small) buffer, I think with the introduction of the separate backend threads, we changed this to "pre-fetch" data from the backend and then deliver it to the client asynchronously.

I could imagine we could limit the amount of such readahead

[nigoroll]

sorry, the references from the bugfix were for #2963

[nigoroll]

bugwash:

• transient should have a reasonable default size
• prefetch should be limited
• move to VIP

[bsdphk]

Bugwash: Not clear how to solve. Move to VIP

[nigoroll]

-> https://github.com/varnishcache/varnish-cache/wiki/VIP-26:-limit-private-prefetch

[anthosz]

Hi,

I just check https://github.com/varnishcache/varnish-cache/wiki/VIP-26:-limit-private-prefetch and it's not clear.

If we try to use pipe on vcl_backend* (to pipe on backend response condition (on Content-Lenght)), it's not allowed (worked in 3.x / but not in 6.3.x).

How can we do to do it now?

Best regards,

[nigoroll]

@anthosz pipe mode is unaffected by this issue, but not recommended because of the lack of control over the backend response.
return(pipe) is only supported from the client side because pipe mode itself is a form of a backend request.

[anthosz]

@anthosz pipe mode is unaffected by this issue, but not recommended because of the lack of control over the backend response.
return(pipe) is only supported from the client side because pipe mode itself is a form of a backend request.

Thx. So how to avoid pass (and avoid Transient storage issue due to memory limit) if we cannot use pipe? :X

The idea is to avoid pass & transient storage usage when there are a big request. I cannot find the solution in V6.

[nigoroll]

you can, from the client side.

[anthosz]

@nigoroll but the Content-Length header is provided by backend (so in vcl_backend_response) :/

The issue is that we cannot pipe in vcl_backend_response or vcl_deliver and we cannot have the Content-Length value in vcl_recv.

[nigoroll]

@anthosz I understand why you want to switch to pipe mode on the backend side, but it's just not possible.
We just need to implement VIP26. If anyone wants to sponsor that, get in touch

[nigoroll]

BTW, there is a poc in #3240

[mnederlof]

We've had this issue too, and made a workaround in VCL code like below to pipe all requests, if the content length header shows that the response exceeds 30GB.

basically, you can restart your request, and go into pipe mode, by chaining through several vcl methods if done creatively...

vcl 4.1;

import std;

sub vcl_backend_response {
    if (std.integer(beresp.http.content-length, 0) > 32212254720) {
        std.log("DEBUG: Fail this error request, as Transient storage might be insufficient.");
        set bereq.http.x-error-reason = "oversize";
        return (error);
    }
}

sub vcl_backend_error {
    if (bereq.http.x-error-reason == "oversize") {
        set beresp.status = 599;
        return (deliver);
    }
}

sub vcl_deliver {
    if (resp.status == 599) {
        std.log("DEBUG: restarting, backend fetch failed due to oversize");
        set req.http.x-pipe-request = "1";
        return (restart);
    }
}

sub vcl_recv {
    # If we have a restart from vcl_deliver, then we immediately go into pipe
    if (req.http.x-pipe-request && req.restarts > 0) {
        return(pipe);
    }
}

[HLeithner]

@mnederlof I do the same thing already but thanks for sharing

[anthosz]

I do the same thing but doesn't works on my side (maybe not related):

All is stable (average usage is ~500MB) during several days and sometimes, there are an overload (+25GB (memory limit) in the transcient storage (out of memory based on panic.show so Varnish crash (6.3)/ cache storage is on FS).

I tried to switch the transcient storage to FS but it take all time all the FS storage & Varnish crash 1 day after.

I don't understand what can cause these behaviors...

[feld]

We've had this issue too, and made a workaround in VCL code like below to pipe all requests, if the content length header shows that the response exceeds 30GB.

basically, you can restart your request, and go into pipe mode, by chaining through several vcl methods if done creatively...

vcl 4.1;

import std;

sub vcl_backend_response {
    if (std.integer(beresp.http.content-length, 0) > 32212254720) {
        std.log("DEBUG: Fail this error request, as Transient storage might be insufficient.");
        set bereq.http.x-error-reason = "oversize";
        return (error);
    }
}

sub vcl_backend_error {
    if (bereq.http.x-error-reason == "oversize") {
        set beresp.status = 599;
        return (deliver);
    }
}

sub vcl_deliver {
    if (resp.status == 599) {
        std.log("DEBUG: restarting, backend fetch failed due to oversize");
        set req.http.x-pipe-request = "1";
        return (restart);
    }
}

sub vcl_recv {
    # If we have a restart from vcl_deliver, then we immediately go into pipe
    if (req.http.x-pipe-request && req.restarts > 0) {
        return(pipe);
    }
}

I'm hitting this bug too and this VCL doesn't work. The best solution I have at the moment is the simple uncachable when content-length is above a limit, but eventually the connection fails and you have to resume the download manually.

[arch-user-france1]

It wont work if you download something. You can't reset the connection without the browser thinking the download failed

Any fix? Is there a way to detect it already in apache or nginx and set no-cache?

[arch-user-france1]

We've had this issue too, and made a workaround in VCL code like below to pipe all requests, if the content length header shows that the response exceeds 30GB.

basically, you can restart your request, and go into pipe mode, by chaining through several vcl methods if done creatively...

vcl 4.1;

import std;

sub vcl_backend_response {
    if (std.integer(beresp.http.content-length, 0) > 32212254720) {
        std.log("DEBUG: Fail this error request, as Transient storage might be insufficient.");
        set bereq.http.x-error-reason = "oversize";
        return (error);
    }
}

sub vcl_backend_error {
    if (bereq.http.x-error-reason == "oversize") {
        set beresp.status = 599;
        return (deliver);
    }
}

sub vcl_deliver {
    if (resp.status == 599) {
        std.log("DEBUG: restarting, backend fetch failed due to oversize");
        set req.http.x-pipe-request = "1";
        return (restart);
    }
}

sub vcl_recv {
    # If we have a restart from vcl_deliver, then we immediately go into pipe
    if (req.http.x-pipe-request && req.restarts > 0) {
        return(pipe);
    }
}

This can't work

It would RESTART it and not simply bypass

[HLeithner]

The workaround only works for good for GET requests but for POST requests in the restart request no POST DATA is provided...

[dridi]

Have you tried std.cache_req_body() to restart POST requests?

https://varnish-cache.org/docs/6.0/reference/vmod_generated.html#func-cache-req-body

[HLeithner]

no didn't tried to but came to the conclusion that POST shouldn't (actually the RFC 2616 forbid to) be cached anyway, so wouldn't it make sense to directly pipe all POST/PUT/DELETE requests?

[gquintard]

I think there's a confusion on what caching means in that context. std.cache_req_body() only makes sure we remember the body for the duration of the request processing, we are not actually catching the response to a POST request.

Also, pipe becomes a pass when using HTTP2 (since we can just give the full connection to a single transaction, so you still need to solve the issue.

[HLeithner]

I understand what cache_req_body() does but the problem it self is that we trigger the same POST quest a second time which actually is a bad idea, since we for example create a second entry.

So for everything which is not a GET we should pipe the request anyway. In case of a GET request we don't have body so we don't need to cache it.

Or I'm wrong? btw. this has nothing to do with http2 at least my backend servers don't use http2.

[dridi]

The responses for POST requests can be cached if the server says so, but yes, care should be taken not to retry a request with side effects.

[gquintard]

this has nothing to do with http2 at least my backend servers don't use http2.

what I'm saying is that if you return(pipe) on a HTTP/2 request, it'll be converted to a return(pass)

[dridi]

Time for discussion to move to a different forum, like a mailing list.

https://varnish-cache.org/lists/mailman/listinfo/varnish-misc