changes.rst

About this document

This document contains notes from the Varnish developers about ongoing development and past versions:

• Developers will note here changes which they consider particularly relevant or otherwise noteworthy
• This document is not necessarily up-to-date with the code
• It serves as a basis for release managers and others involved in release documentation
• It is not rendered as part of the official documentation and thus only available in ReStructuredText (rst) format in the source repository and -distribution.

Official information about changes in releases and advise on the upgrade process can be found in the doc/sphinx/whats-new/ directory, also available in HTML format at http://varnish-cache.org/docs/trunk/whats-new/index.html and via individual releases. These documents are updated as part of the release process.

Varnish Cache 7.2.1 (2022-11-08)

• Attempts to mark well-known headers like Content-Length and Host hop-by-hop through a Connection-header will now cause a 400 "Bad request" response. (VSV00010)
• Apply the same character set rules to HTTP/2 pseudo-headers as is done on the corresponding HTTP/1 request-line field parsing. (VSV00011)
• Fix a problem related to logging strands and unfortunate alignment, which would result in a panic. (3856)

Varnish Cache 7.2.0 (2022-09-15)

• Functions VRT_AddVDP(), VRT_AddVFP(), VRT_RemoveVDP() and VRT_RemoveVFP() are deprecated.

• Cookie headers generated by vmod_cookie no longer have a spurious trailing semi-colon (';') at the end of the string. This could break VCL relying on the previous incorrect behavior.

• The SessClose and BackendClose reason rx_body, which previously output Failure receiving req.body, has been rewritten to Failure receiving body.

• Prototypical Varnish Extensions (VEXT). Similar to VMODs, a VEXT is loaded by the cache process. Unlike VMODs that have the combined lifetime of all the VCLs that reference them, a VEXT has the lifetime of the cache process itself. There are no built-in extensions so far.

• The VCC (compilation) process no longer loads VMODs with dlopen(3) to collect their metadata.

• Stevedore initialization via the .init() callback has been moved to the worker process.

• The parameter tcp_keepalive_time is supported on MacOS.

• Duration parameters can optionally take a unit, with the same syntax as duration units in VCL. Example: param.set default_grace 1h.

• Calls to VRT_CacheReqBody() and std.cache_req_body from outside client vcl subs now fail properly instead of triggering an assertion failure (3846).

• New "B" string for the package branch in VCS_String(). For the 7.2.0 version, it would yield the 7.2 branch.

• The Varnish version and branch are available in varnishtest through the ${pkg_version} and ${pkg_branch} macros.

• New ${topsrc} macro in varnishtest -i mode.

• New process pNAME -match-text command in varnishtest to expect text matching a regular expression on screen.

• New filewrite [-a] command in varnishtest to put or append a string into a file.

• The new vcc_feature bits parameter replaces previous vcc_* boolean parameters. The latter still exist as deprecated aliases.

• The -k option from varnishlog is now supported by varnishncsa.

• New functions std.now() and std.timed_call() in vmod_std.

• New MAIN.shm_bytes counter.

• A req.http.via header is set before entering vcl_recv. Via headers are generated using the server.identity value. It defaults to the host name and can be turned into a pseudonym with the varnishd -i option. Via headers are appended in both directions, to work with other hops that may advertise themselves.

• A resp.http.via header is no longer overwritten by varnish, but rather appended to.

• The server.identity syntax is now limited to a "token" as defined in the HTTP grammar to be suitable for Via headers.

• In varnishtest a Varnish instance will use its VTC instance name as its instance name (varnishd -i) by default for predictable Via headers in test cases.

• VMOD and VEXT authors can use functions from vnum.h.

• Do not filter pseudo-headers as regular headers (VSV00009 / 3830).

• The termination rules for WRK_BgThread() were relaxed to allow VMODs to use it.

• (struct worker).handling has been moved to the newly introduced struct wrk_vpi and replaced by a pointer to it, as well as (struct vrt_ctx).handling has been replaced by that pointer.

  struct wrk_vpi is for state at the interface between VRT and VGC and, in particular, is not const as struct vrt_ctx aka VRT_CTX.

• Panics now contain information about VCL source files and lines.

• The Begin log record has a 4th field for subtasks like ESI sub-requests.

• The -E option for log utilities now works as documented, with any type of sub-task based on the Begin[4] field. This covers ESI like before, and sub-tasks spawned by VMODs (provided that they log the new field).

• No more req.http.transfer-encoding for ESI sub-requests.

• New tools/coccinelle/vcocci.sh refactoring script for internal use.

• The thread pool reserve is now limited to tasks that can be queued. A backend background fetch is no longer eligible for queueing. It would otherwise slow a grace hit down significantly when thread pools are saturated.

• The unused fetch_no_thread counter was renamed to bgfetch_no_thread because regular backend fetch tasks are always scheduled.

• The macros FEATURE(), EXPERIMENT(), DO_DEBUG(), MGT_FEATURE(), MGT_EXPERIMENT(), MGT_DO_DEBUG() and MGT_VCC_FEATURE() now return a boolean value (0 or 1) instead of the (private) flag value.

• There is a new contrib/ directory in the Varnish source tree. The first contribution is a varnishstatdiff script.

• A regression in the transport code led MAIN.client_req to be incremented for requests coming back from the waiting list, it was fixed. (3841)

Varnish Cache 7.1.0 (2022-03-15)

• The cookie.format_rfc1123() function was renamed to cookie.format_date(), and the former was retained as a deprecated alias.

• The VCC file $Alias stanza has been added to support vmod alias functions/methods.

• VCC now supports alias symbols.

• There is a new experimental parameter that is identical to the feature parameter, except that it guards features that may not be considered complete or stable. An experimental feature may be promoted to a regular feature or dropped without being considered a breaking change.

• ESI includes now support the onerror="continue" attribute of <esi:include/> tags.

  The +esi_include_onerror feature flag controls if the attribute is honored: If enabled, failure of an include stops ESI processing unless the onerror="continue" attribute was set for it.

  The feature flag is off by default, preserving the existing behavior to continue ESI processing despite include failures.

• The deprecated sub-argument of the -l option was removed, it is now a shorthand for the vsl_space parameter only.

• The -T, -M and -P command line options can be used multiple times, instead of retaining only the last occurrence.

• The debug.xid CLI command has been extended to also set and query the VXID cache chunk size.

• The vtc.barrier_sync() VMOD function now also works in vcl_init

• The abort command in the logexpect facility of varnishtest can now be used to trigger an abort() to help debugging the vsl client library code.

• The vtc.vsl() and vtc.vsl_replay() functions have been added to the vtc vmod to generate arbitraty log lines for testing.

• The limit of the vsl_reclen parameter has been corrected.

• Varnish now closes client connections correctly when request body processing failed.

• Filter init methods of types vdp_init_f and vfp_init_f gained a VRT_CTX argument.

• The param.set CLI command accepts a -j option. In this case the JSON output is the same as param.show -j of the updated parameter.

• A new cc_warnings parameter contains a subset of the compiler flags extracted from cc_command, which in turn grew new expansions:

  • %d: the raw default cc_command
  • %D: the expanded default cc_command
  • %w: the cc_warnings parameter
  • %n: the working directory (-n option)

• For return(pipe), the backend transactions now emit a Start timestamp and both client and backend transactions emit the Process timestamp.

• http_IsHdr() is now exposed as part of the strict ABI for VMODs.

• The req.transport VCL variable has been added, which returns "HTTP/1" or "HTTP/2" as appropriate.

• The vtc.workspace_reserve() VMOD function now zeroes memory.

• Parameter aliases have been added to facilitate parameter deprecation.

• Two bugs in the catflap facility have been fixed which could trigger panics due to the state pointer not being cleared. (3752, 3755)

• It is now possible to assign to a BODY variable either a STRING type or a BLOB.

• When the vcl.show CLI command is invoked without a parameter, it now defaults to the active VCL.

• The reporting of logexpect events in varnishtest was rearranged for readability.

• Workspace debugging as enabled by the +workspace debug flag is now logged with the corresponding transaction.

• VMODs should now register and unregister fetch and delivery filters with VRT_AddFilter() and VRT_RemoveFilter().

• HSH_purge() has been rewritten to properly handle concurrent purges on the same object head.

• VSL_WriteOpen(), varnishlog and varnishncsa have been changed to support writing to stdout with -w - when not in daemon mode.

• In VSL, the case has been optimized that the space remaining in a buffer is close to vsl_reclen.

• std.ip() has been changed to always return a valid (bogo ip) fallback if the fallback argument is invalid.

• New VCL variables {req,req_top,resp,bereq,beresp,obj}.time have been added to track when the respective object was born.

• VRT_StaticDirector() has been added to mark directors with VCL lifetime, to avoid the overhead of reference counting.

• Dynamic backends are now reference-counted, and VMOD authors must explicitly track assignments with VRT_Assign_Backend().

• Varnish will use libunwind by default when available at configure time, the --without-unwind configure flag can prevent this and fall back to libexecinfo to generate backtraces.

• A new debug.shutdown.delay command is available in the Varnish CLI for testing purposes.

• New utility macros vmin[_t], vmax[_t] and vlimit[_t] available in vdef.h.

• The macros TOSTRAND(s) and TOSTRANDS(x, ...) have been added to create a struct strands * (intended to be used as a VCL_STANDS) from a single string s or x strings, respectively.

  Note that the macros create a compund literal whose scope is the enclosing block. Their value must thus only be used within the same block (it can be passed to called functions) and must not be returned or referenced for use outside the enclosing block.

  As before, VRT_AllocStrandsWS() or VRT_StrandsWS() must be used to create VCL_STRANDS with task scope for use outside the current block.

• A bug in the backend connection handling code has been fixed which could trigger an unwarranted assertion failure (3664).

• std.strftime() has been added.

• Lck_CondWait() has lost the timeout argument and now waits forever. Lck_CondWaitUntil() and Lck_CondWaitTimeout() have been added to wait on a condition variable until some point in time or until a timeout expires, respectively.

• All mutex locks in core code have been given the PTHREAD_MUTEX_ERRORCHECK attribute.

• Host and Content-Length header checks have been moved to protocol independent code and thus implicitly extended to HTTP2.

• A potential race on busy objects has been closed.

• Use of the ObjGetSpace() for synthetic objects has been fixed to support stevedores returning less space than requested (as permitted by the API).

• The FINI_OBJ() macro has been added to standardize the common pattern of zeroing a mini object and clearing a pointer to it.

• The deprecated vsm_space parameter was removed.

• The varnishtest err_shell commando has been removed after having been deprecated since release 5.1.0.

Varnish Cache 7.0.1 (2021-11-23)

• An assertion failure has been fixed which triggered when matching bans on non-existing headers (3706).

• A VCL compilation issue has been fixed when calling builtin functions directly (3719).

• It is now again possible to concatenate static strings to produce combined strings of type VCL_REGEX (3721).

• An issue has been fixed that would cause the VCL dependency checker to incorrectly flag VCLs as dependants of other VCLs when using labels, preventing them from being discarded (3734).

• VCLs loaded through CLI or the use of startup CLI scripts (-I option to varnishd) will, when no active VCL has previously been set, no longer automatically set the first VCL loaded to the active VCL. This prevents situations where it was possible to make a cold VCL the active VCL (3737).

• There is now a configure build-time requirement on working SO_RCVTIMEO and SO_SNDTIMEO socket options.

  We no longer check whether they effectively work, so the SO_RCVTIMEO_WORKS feature check has been removed from varnishtest.

• The socket option inheritance checks now correctly identifies situations where UDS and TCP listening sockets behave differently, and are no longer subject to the order the inheritance checks happens to be executed (3732).

• IPv6 listen endpoint address strings are now printed using brackets.

Varnish Cache 7.0.0 (2021-09-15)

• Added convenience vrt_null_strands and vrt_null_blob constants.
• New VCL flag syntax foo <name> +bar -baz { ... }, starting with ACL flags log, pedantic and table.
• ACLs no longer produce VSL VCL_acl records by default, this must be explicitly enabled with acl <name> +log { ... }.
• ACLs can be compiled into a table format, which runs a little bit slower, but compiles much faster for large ACLs.
• ACLs default to pedantic which is now a per-ACL feature flag.
• New glob flag for VCL include (3193).
• The maximum number of headers for a request or a response in varnishtest was increased to 64.
• The backend lock class from struct backend was moved to struct director and renamed accordingly.
• New %{sec,msec,usec,msec_frac,usec_frac}t formats in varnishncsa.
• vstrerror() was renamed to VAS_errtxt().
• New varnishncsa -j option to format for JSON (3595).
• To skip a test in the presence of a feature instead of it absence, a new feature !<name> syntax was added to varnishtest.
• Accept-Ranges headers are no longer generated for passed objects, but must either come from the backend or be created in vcl_deliver{} (3251).
• The busyobj do_pass flag is gone in favor of uncacheable.
• The objcore flag ABANDON was renamed to CANCEL.
• 'Scientific Notation' numbers like 6.62607004e-34 are no longer supported in VCL. (The preparation of RFC8941 made it clear that there are neither reason nor any need to support scientific notation in context of HTTP headers.
• New tunnel command in varnishtest to gain the ability to shape traffic between two peers without having to change their implementation.
• Global VCL symbols can be defined after use (3555).
• New req.hash_ignore_vary flag in VCL.
• varnishtest can register macros backed by functions, which is the case for ${date} and the brand new ${string,<action>[,<args>...]} macro (3627).
• Migration to pcre2 with extensive changes to the VRE API, parameters renamed to pcre2_match_limit and pcre2_depth_limit, and the addition of a new pcre2_jit_compilation parameter. The varnishtest undocumented feature check pcre_jit is gone (3635). This change is transparent at the VRT layer and only affects direct VRE consumers.
• New inverted mode in vtc-bisect.sh to find the opposite of regressions.
• The default values for workspace_client, workspace_backend and vsl_buffer on 64bit systems were increased to respectively 96kB, 96kB and 16kB (3648).
• The deprecated WS_Inside() was replaced with WS_Allocated() and WS_Front() was removed.
• VCL header names can be quoted, for example req.http."valid.name".
• Added VRT_UnsetHdr() and removed vrt_magic_string_unset.
• Removed depcreated STRING_LIST in favor of STRANDS. All functions that previously took a STRING_LIST had const char *, ... arguments, they now take const char *, VCL_STRANDS arguments. The magic cookie vrt_magic_string_end is gone and VRT_CollectStrands() was renamed to VRT_STRANDS_string().
• The default value for thread_pool_stack was increased to 80kB for 64bit systems and 64kB for 32bit to accomodate the PCRE2 jit compiler.
• Removed deprecated VSB_new() and VSB_delete(), which resulted in a major soname bump of libvarnishapi to 3.0.0, instead of the 2.7.0 version initially planned.
• The default workdir (the default -n argument) is now /var/run instead of ${prefix}/var (3672). Packages usually configure this to match local customs.
• The minimum session_workspace is now 384 bytes
• Emit minimal 500 response if vcl_synth fails (3441).
• New --enable-coverage configure flag, and renovated sanitizer setup.
• New feature checks in varnishtest: sanitizer, asan, lsan, msan, ubsan and coverage.
• New --enable-workspace-emulator configure flag to swap the worksapce implementation with a sparse one ideal for fuzzing (3644).
• Strict comparison of items from the HTTP grammar (3650).
• New request body h2 window handling using a buffer to avoid stalling an entire h2 session until the relevant stream starts consuming DATA frames. As a result the minimum value for h2_initial_window_size is now 65535B to avoid running out of buffer with a negative window that was simpler to not tolerate, and a new h2_rxbuf_storage parameter was added (3661).
• SLT_Hit now includes streaming progress when relevant.
• The http_range_support adds consistency checks for pass transactions (3673).
• New VNUM_uint() and VNUM_hex() functions geared at token parsing.

Varnish Cache 6.6.0 (2021-03-15)

• Body bytes accounting has been fixed to always represent the number of bodybytes moved on the wire, exclusive of protocol-specific overhead like HTTP/1 chunked encoding or HTTP/2 framing.

  This change affects counters like

  • MAIN.s_req_bodybytes,
  • MAIN.s_resp_bodybytes,
  • VBE.*.*.bereq_bodybytes and
  • VBE.*.*.beresp_bodybytes

  as well as the VSL records

  • ReqAcct,
  • PipeAcct and
  • BereqAcct.

• VdpAcct log records have been added to output delivery filter (VDP) accounting details analogous to the existing VfpAcct. Both tags are masked by default.

• Many filter (VDP/VFP) related signatures have been changed:

  • vdp_init_f
  • vdp_fini_f
  • vdp_bytes_f
  • VDP_bytes()

  as well as struct vdp_entry and struct vdp_ctx

  VFP_Push() and VDP_Push() are no longer intended for VMOD use and have been removed from the API.

• The VDP code is now more strict about VDP_END, which must be sent down the filter chain at most once.

• Core code has been changed to ensure for most cases that VDP_END gets signaled with the object's last bytes, rather than with an extra zero-data call.

• Reason phrases for more HTTP Status codes have been added to core code.

• Connection pooling behavior has been improved with respect to Connection: close (3400, 3405).

• Handling of the Keep-Alive HTTP header as hop-by-hop has been fixed (3417).

• Handling of hop-by-hop headers has been fixed for HTTP/2 (3416).

• The stevedore API has been changed:

  • OBJ_ITER_FINAL has been renamed to OBJ_ITER_END
  • ObjExtend() signature has been changed to also cover the ObjTrimStore() use case and
  • ObjTrimStore() has been removed.

• The verrno.h header file has been removed and merged into vas.h

• The connection close reason has been fixed to properly report SC_RESP_CLOSE / resp_close where previously only SC_REQ_CLOSE / req_close was reported.

• Unless the new validate_headers feature is disabled, all newly set headers are now validated to contain only characters allowed by RFC7230. A (runtime) VCL failure is triggered if not (3407).

• VRT_ValidHdr() has been added for vmods to conduct the same check as the validate_headers feature, for example when headers are set by vmods using the cache_http.c Functions like http_ForceHeader() from untrusted input.

• The shard director now supports reconfiguration (adding/removing backends) of several instances without any special ordering requirement.

• Calling the shard director .reconfigure() method is now optional. If not called explicitly, any shard director backend changes are applied at the end of the current task.

• Shard director Error log messages with (notice) have been turned into Notice log messages.

• All shard Error and Notice messages now use the unified prefix vmod_directors: shard %s.

• In the shard director, use of parameter sets with resolve=NOW has been fixed.

• Performance of log-processing tools like varnishlog has been improved by using mmap() if possible when reading from log files.

• An assertion failure has been fixed which could be triggered when a request body was used with restarts (3433, 3434).

• A signal handling bug in the Varnish Utility API (VUT) has been fixed which caused log-processing utilities to perform poorly after a signal had been received (3436).

• The client.identity variable is now accessible on the backend side.

• Client and backend finite state machine internals (enum req_step and enum fetch_step) have been removed from cache.h.

• Three new Timestamp VSL records have been added to backend request processing:

  • The Process timestamp after return(deliver) or return(pass(x)) from vcl_backend_response,
  • the Fetch timestamp before a backend connection is requested and
  • the Connected timestamp when a connection to a regular backend (VBE) is established, or when a recycled connection was selected for reuse.

• The VRT backend interface has been changed:

  • struct vrt_endpoint has been added describing a UDS or TCP endpoint for a backend to connect to.

    Endpoints also support a preamble to be sent with every new connection.

  • This structure needs to be passed via the endpoint member of struct vrt_backend when creating backends with VRT_new_backend() or VRT_new_backend_clustered().

• VRT_Endpoint_Clone() has been added to facilitate working with endpoints.

• The variables bereq.is_hitpass and bereq.is_hitmiss have been added to the backend side matching req.is_hitpass and req.is_hitmiss on the client side.

• The set_ip_tos() function from the bundled std vmod now sets the IPv6 Taffic Class (TCLASS) when used on an IPv6 connection.

• A bug has been fixed which could lead to varnish failing to start after updates due to outdated content of the vmod_cache directory (3243).

• An issue has been addressed where using VCL with a high number of literal strings could lead to prolonged c-compiler runtimes since Varnish-Cache 6.3 (3392).

• The MAIN.esi_req counter has been added as a statistic of the number of ESI sub requests created.

• The vcl.discard CLI command can now be used to discard more than one VCL with a single command, which succeeds only if all given VCLs could be discarded (atomic behavior).

• The vcl.discard CLI command now supports glob patterns for vcl names.

• The vcl.deps CLI command has been added to output dependencies between VCLs (because of labels and return(vcl) statements).

• The FetchError log message Timed out reusing backend connection has been renamed to first byte timeout (reused connection) to clarify that it is emit for effectively the same reason as first byte timeout.

• Long strings in VCL can now also be denoted using """ ... """ in addition to the existing {" ... "}.

• The pdiff() function declaration has been moved from cache.h to vas.h.

• The interface for private pointers in VMODs has been changed:

  • The free pointer in struct vmod_priv has been replaced with a pointer to struct vmod_priv_methods, to where the pointer to the former free callback has been moved as the fini member.
  • The former free callback type has been renamed from vmod_priv_free_f to vmod_priv_fini_f and as gained a VRT_CTX argument

• The MAIN.s_bgfetch counter has been added as a statistic on the number of background fetches issues.

• Various improvements have been made to the varnishtest facility:

  • the loop keyword now works everywhere
  • HTTP/2 logging has been improved
  • Default HTTP/2 parameters have been tweaked (3442)
  • Varnish listen address information is now available by default in the macros ${vNAME_addr}, ${vNAME_port} and ${vNAME_sock}. Macros by the names ${vNAME_SOCKET_*} contain the address information for each listen socket as created with the -a argument to varnishd.
  • Synchronization points for counters (VSCs) have been added as varnish vNAME -expect PATTERN OP PATTERN
  • varnishtest now also works with IPv6 setups
  • feature ipv4 and feature ipv6 can be used to control execution of test cases which require one or the other protocol.
  • haproxy arguments can now be externally provided through the HAPROXY_ARGS variable.
  • logexpect now supports alternatives with the expect ? ... syntax and negative matches with the fail add ... and fail clear syntax.
  • The overall logexpect match expectation can now be inverted using the -err argument.
  • Numeric comparisons for HTTP headers have been added: -lt, -le, -eq, -ne, -ge, -gt
  • rxdata -some has been fixed.

• The ban_cutoff parameter now refers to the overall length of the ban list, including completed bans, where before only non-completed ("active") bans were counted towards ban_cutoff.

• A race in the round-robin director has been fixed which could lead to backend requests failing when backends in the director were sick (3473).

• A race in the probe management has been fixed which could lead to a panic when VCLs changed temperature in general and when vcl.discard was used in particular (3362).

• A bug has been fixed which lead to counters (VSCs) of backends from cold VCLs being presented (3358).

• A bug in varnishncsa has been fixed which could lead to it crashing when header fields were referenced which did not exist in the processed logs (3485).

• For failing PROXY connections, SessClose now provides more detailed information on the cause of the failure.

• The std.ban() and std.ban_error() functions have been added to the std vmod, allowing VCL to check for ban errors.

• Use of the ban() built-in VCL command is now deprecated.

• The source tree has been reorganized with all vmods now moved to a single vmod directory.

• vmodtool.py has been improved to simplify Makefiles when many VMODs are built in a single directory.

• The VSA_getsockname() and VSA_getpeername() functions have been added to get address information of file descriptors.

• varnishd now supports the -b none argument to start with only the builtin VCL and no backend at all (3067).

• Some corner cases of IPv6 support in varnishd have been fixed.

• vcl_pipe {}: return(synth) and vmod private state support have been fixed. Trying to use std.rollback() from vcl_pipe now results in VCL failure (3329, 3330, 3385).

• The bereq.xid variable is now also available in vcl_pipe {}

• The VRT_priv_task_get() and VRT_priv_top_get() functions have been added to VRT to allow vmods to retrieve existing PRIV_TASK / PRIV_TOP private pointers without creating any.

• varnishstat now avoids display errors of gauges which previously could underflow to negative values, being displayed as extremely high positive values.

  The -r option and the r key binding have been added to return to the previous behavior. When raw mode is active in varnishstat interactive (curses) mode, the word RAW is displayed at the right hand side in the lower status line.

• The VSC_IsRaw() function has been added to libvarnishapi to query if a gauge is being returned raw or adjusted.

• The busy_stats_rate feature flag has been added to ensure statistics updates (as configured using the thread_stats_rate parameter) even in scenarios where worker threads never run out of tasks and may remain forever busy.

• ExpKill log (VSL) records are now masked by default. See the vsl_mask parameter.

• A bug has been fixed which could lead to panics when ESI was used with ESI-aware VMODs were used because PRIV_TOP vmod private state was created on a wrong workspace (3496).

• The VCL_REGEX data type is now supported for VMODs, allowing them to use regular expression literals checked and compiled by the VCL compiler infrastructure.

  Consequently, the VRT_re_init() and VRT_re_fini() functions have been removed, because they are not required and their use was probably wrong anyway.

• The filter_re, keep_re and get_re functions from the bundled cookie vmod have been changed to take the VCL_REGEX type. This implies that their regular expression arguments now need to be literal, whereas before they could be taken from some other variable or function returning VCL_STRING.

  Note that these functions never actually handled _dynamic_ regexen, the string passed with the first call was compiled to a regex, which was then used for the lifetime of the respective VCL.

• The %{X}T format has been added to varnishncsa, which generalizes %D and %T, but also support milliseconds (ms) output.

• Error handling has been fixed when vmod functions/methods with PRIV_TASK arguments were wrongly called from the backend side (3498).

• The varnishncsa -E argument to show ESI requests has been changed to imply -c (client mode).

• Error handling and performance of the VSL (shared log) client code in libvarnishapi have been improved (3501).

• varnishlog now supports the -u option to write to a file specified with -w unbuffered.

• Comparisons of numbers in VSL queries have been improved to match better the behavior which is likely expected by users who have not read the documentation in all detail (3463).

• A bug in the ESI code has been fixed which could trigger a panic when no storage space was available (3502).

• The resp.proto variable is now read-only as it should have been for long.

• VTCP_open() has been fixed to try all possible addresses from the resolver before giving up (3509). This bug could cause confusing error messages (3510).

• VRT_synth_blob() and VRT_synth_strands() have been added. The latter should now be used instead of VRT_synth_page().

• The VCL_SUB data type is now supported for VMODs to save references to subroutines to be called later using VRT_call(). Calls from a wrong context (e.g. calling a subroutine accessing req from the backend side) and recursive calls fail the VCL.

  See VMOD - Varnish Modules in the Reference Manual.

VMOD functions can also return the VCL_SUB data type for calls from VCL as in call vmod.returning_sub();.

• VRT_check_call() can be used to check if a VRT_call() would succeed in order to avoid the potential VCL failure in case it would not.

  It returns NULL if VRT_call() would make the call or an error string why not.

• VRT_handled() has been added, which is now to be used instead of access to the handling member of VRT_CTX.

• The session close reason logging/statistics for HTTP/2 connections have been improved (3393)

• varnishadm now has the -p option to disable readline support for use in scripts and as a generic CLI connector.

• A log (VSL) Notice record is now emitted whenever more than vary_notice variants are encountered in the cache for a specific hash. The new vary_notice parameter defaults to 10.

• The modulus operator % has been added to VCL.

• return(retry) from vcl_backend_error {} now correctly resets beresp.status and beresp.reason (3525).

• Handling of the gunzip filter with ESI has been fixed (3529).

• A bug where the threads_limited counter could be increased without reason has been fixed (3531).

• All varnish tools using the VUT library utilities for argument processing now support the --optstring argument to return a string suitable for use with getopts from shell scripts.

• An issue with high CPU consumption when the maximum number of threads was reached has been fixed (2942, 3531)

• HTTP/2 streams are now reset for filter chain (VDP) errors.

• The task priority of incoming connections has been fixed.

• An issue has been addressed where the watchdog facility could misfire when tasks are queued.

• The builtin VCL has been reworked: VCL code has been split into small subroutines, which custom VCL can prepend custom code to.

  This allows for better integration of custom VCL and the built-in VCL and better reuse.

Varnish Cache 6.5.1 (2020-09-25)

• Bump the VRT_MAJOR_VERSION from 11 to 12, to reflect the API changes that went into the 6.5.0 release. This step was forgotten for that release.

Varnish Cache 6.5.0 (2020-09-15)

[ABI] marks potentially breaking changes to binary compatibility.

[API] marks potentially breaking changes to source compatibility (implies [ABI]).

• varnishstat now has a help screen, available via the h key in curses mode

• The initial varnishstat verbosity has been changed to ensure any fields specified by the -f argument are visible (2990)

• Fixed handling of out-of-workspace conditions after vcl_backend_response and vcl_deliver during filter initialization (3253, 3241)

• PRIV_TOP is now thread-safe to support parallel ESI implementations

• varnishstat JSON format (-j option) has been changed:

  • on the top level, a version identifier has been introduced, which will be used to mark future breaking changes to the JSON formatting. It will not be used to mark changes to the counters themselves.

    The new version is 1.

  • All counters have been moved down one level to the counters object.

• VSA_BuildFAP() has been added as a convenience function to build a struct suckaddr

• Depending on the setting of the new vcc_acl_pedantic parameter, VCC now either emits a warning or fails if network numbers used in ACLs do not have an all-zero host part.

  For vcc_acl_pedantic off, the host part is fixed to all-zero and that fact logged with the ACL VSL tag.

• Fixed error handling during object creation after vcl_backend_response (3273)

• obj.can_esi has been added to identify if the response can be ESI processed (3002)

• resp.filters now contains a correct value when the auto-determined filter list is read (3002)

• It is now a VCL (runtime) error to write to resp.do_* and beresp.do_* fields which determine the filter list after setting resp.filters and beresp.filters, respectively

• Behavior for 304 responses was changed not to update the Content-Encoding response header of the stored object.

• [ABI] struct vfp_entry and struct vdp_ctx changed

• [API] VSB_QUOTE_GLOB, which was prematurely added to 6.4, has been removed again.

• [API] Add VDP_END action for delivery processors, which has to be sent with or after the last buffer.

• Respect the administrative health for "real" (VBE) backends (3299)

• Fixed handling of illegal (internal) four-digit response codes and with HTTP/2 (3301)

• Fixed backend connection pooling of closed connections (3266)

• Added the .resolve method for the BACKEND type to resolve (determine the "real" backend) a director.

• Improved vmodtool support for out-of-tree builds

• Added VJ_unlink() and VJ_rmdir() jail functions

• Fixed workdir cleanup (3307)

• Added JAIL_MASTER_SYSTEM jail level

• The Varnish Jail (least privileges) code for Solaris has been largely rewritten. It now reduces privileges even further and thus should improve the security of Varnish on Solaris even more.

• The Varnish Jail for Solaris now accepts an optional worker= argument which allows to extend the effective privilege set of the worker process.

• The shard director and shard director parameter objects should now work in vcl_pipe {} like in vcl_backend_* {} subs.

• For a failure in vcl_recv {}, the VCL state engine now returns right after return from that subroutine. (3303)

• The shard director now supports weights by scaling the number of replicas of each backend on the consistent hashing ring

• Fixed a race in the cache expiry code which could lead to a panic (2999)

• Added VRE_quote() to facilitate building literal string matches with regular expressions.

• The BackendReuse VSL (log) tag has been retired and replaced with BackendClose, which has been changed to contain either close or recycle to signify whether the connection was closed or returned to a pool for later reuse.

• BackendOpen VSL entries have been changed to contain reuse or connect in the last column to signify whether the connection was reused from a pool or newly opened.

• std.rollback() of backend requests with return(retry) has been fixed (3353)

• FetchError logs now differentiate between No backend and "none resolved" as Director %s returned no backend

• Added VRT_DirectorResolve() to resolve a director

• Improved VCC handling of symbols and, in particular, type methods

• Fixed use of the shard director from vcl_pipe {} (3361)

• Handle recursive use of vcl include (3360)

• VCL: Added native support for BLOBs in structured fields notation (:<base64>:)

• Fixed handling of the Connection: header when multiple instances of the named headers existed.

• Added support for naming PRIV_ arguments to vmod methods/functions

• The varnish binary heap implementation has been renamed to use the VBH_ prefix, complemented with a destructor and added to header files for use with vmods (via include of vbh.h).

• A bug in vmod_blob for base64 decoding with a length argument and non-padding decoding has been fixed (3378)

• Added VRT_BLOB_string() to vrt.h

• VSB support for dynamic vs. static allocations has been changed:

  For dynamic allocations use:

  VSB_new_auto() + VSB_destroy()
  

  For preexisting buffers use:

  VSB_init() + VSB_fini()
  

  VSB_new() + VSB_delete() are now deprecated.

• std.blobread() has been added

• New MAIN.beresp_uncacheable and MAIN.beresp_shortlived counters have been added.

• The I, X and R arguments have been added to the VSC API and varnishstat for inclusion, exclusion and required glob patterns on the statistic field names. (3394)

  • Added the missing VSC_OPT_f macro and the new VSC_OPT_I and VSC_OPT_X to libvarnishapi headers.
  • Added -I and -X options to varnishstat.

• Overhaul of the workspace API

  • The previously deprecated WS_Reserve() has been removed
  • The signature of WS_Printf() has been changed to return const char * instead of void * (we do not consider this a breaking change).
  • Add WS_ReservationSize()
  • WS_Front() is now deprecated and replaced by WS_Reservation()

• Handle a workspace overflow in VRY_Validate() (3319)

• Fixed the backend probe .timeout handling for "dripping" responses (3402)

• New VARNISH_VMODS_GENERATED() macro in varnish.m4.

• Prevent pooling of a Connection: close backend response.

  When this header is present, be it sent by the backend or added in vcl_backend_response {}, varnish closes the connection after the current request. (3400)

Varnish Cache 6.4.0 (2020-03-16)

• The MAIN.sess_drop counter is gone.

• New configure switch: --with-unwind. Alpine linux appears to offer a libexecinfo implementation that crashes when called by Varnish, this offers the alternative of using libunwind instead.

• backend none was added for "no backend".

• std.rollback(bereq) is now safe to use, fixed bug 3009

• Fixed varnishstat, varnishtop, varnishhist and varnishadm handling INT, TERM and HUP signals (bugs 3088 and 3229)

• The hash algorithm of the hash director was changed, so backend selection will change once only when upgrading. Users of the hash director are advised to consider using the shard director, which, amongst other advantages, offers more stable backend selection through consistent hashing.

• Log records can safely have empty fields or fields containing blanks if they are delimited by "double quotes". This was applied to SessError and Backend_health.

• It is now possible for VMOD authors to customize the connection pooling of a dynamic backend. A hash is now computed to determine uniqueness and a backend declaration can contribute arbitrary data to influence the pool.

• The option varnishtest -W is gone, the same can be achieved with varnishtest -p debug=+witness. A witness.sh script is available in the source tree to generate a graphviz dot file and detect potential lock cycles from the test logs.

• The Process timestamp for vcl_synth {} was wrongly issued before the VCL subroutine, now it gets emitted after VCL returns for consistency with vcl_deliver {}.

• Latencies for newly created worker threads to start work on congested systems have been improved.

• VRB_Iterate() signature has changed

• VRT_fail() now also works from director code

• Deliberately closing backend requests through return(abandon), return(fail) or return(error) is no longer accounted as a fetch failure

• Fixed a bug which could cause probes not to run

• The if-range header is now handled, allowing clients to conditionally request a range based on a date or an ETag.

• Introduced struct reqtop to hold information on the ESI top request and PRIV_TOP, fixed regression 3019

• Allow numerical expressions in VCL to be negative / negated

• Add vi-stype CTRL-f / CTRL-b for page down/up to interactive varnishstat

• Fixed wrong handling of an out-of-workspae condition in the proxy vmod and in the workspace allocator, bug 3131

• Raised the minimum for the vcl_cooldown parameter to 1s to fix bug 3135

• Improved creation of additional threads when none are available

• Fixed a race between director creation and the backend.list CLI command - see bug 3094

• Added error handling to avoid panics for workspace overflows during session attribute allocation - bug 3145

• Overloaded the += operator to also append to headers

• Fixed set *.body commands.

• Fixed status for truncated CLI responses, bug 3038

• New or improved Coccinelle semantic patches that may be useful for VMOD or utilities authors.

• Output VCC warnings also for VCLs loaded via the varnishd -f option, see bug 3160

• Improved fetch error handling when stale objects are present in cache, see bug 3089

• Added a Notice VSL tag (used for varnishlog logging)

• Always refer to sub as subroutine in the documentation and error messages to avoid confusion with other terms.

• New pid command in the Varnish CLI, to get the master and optionally cache process PIDs, for example from varnishadm.

• Fixed a race that could result in a partial response being served in its entirety when it is also compressed with gzip.

• Fixed session close reason reporting and accounting, added rx_close_idle counter for separate accounting when timeout_idle is reached. Also, send_timeout is no longer reported as "remote closed".

• Fixed handling of request bodies for backend retries

• Fix deadlocks when the maximum number of threads has been reached, in particular with http/2, see 2418

• Add more vcl control over timeouts with sess.timeout_linger, sess.send_timeout and sess.idle_send_timeout

• Fix panics due to missing EINVAL handling on MacOS, see 1853

• Added VSLs() and VSLbs() functions for logging STRANDS to VSL

• Fixed cases where a workspace overflow would not result in a VCL failure, see 3194

• Added WS_VSB_new() / WS_VSB_finish() for VSBs on workspaces

• Imported vmod_cookie from varnish_modules

  The previously deprecated function cookie.filter_except() has been removed during import. It was replaced by cookie.keep()

• body_status and req_body_status have been collapsed into one type. In particular, the REQ_BODY_* enums now have been replaced with BS_*.

• Fixed an old regression of the Age: header for passes, see bug 3221

• Added VRT_AllocStrandsWS() as a utility function to allocate STRANDS on a workspace.

• Reduced compile time of vcl_init{} / vcl_fini{} with gcc, added v_dont_optimize attribute macro

• Fixed a case where send_timeout would have no effect when streaming from a backend fetch, see bug 3189

  NOTE Users upgrading varnish should re-check send_timeout with respect to long pass and streaming fetches and watch out for increased session close rates.

• Added VSB_tofile() to libvarnishapi, see 3238

Varnish Cache 6.3.0 (2019-09-15)

In addition to a significant number of bug fixes, these are the most important changes in 6.3:

• The Host: header is folded to lower-case in the builtin_vcl.
• Improved performance of shared memory statistics counters.
• Synthetic objects created from vcl_backend_error {} now replace existing stale objects as ordinary backend fetches would, unless:
  • abandoning the bereq or
  • leaving vcl_backend_error {} with return (deliver) and ``beresp.ttl == 0s or
  • there is a waitinglist on the object, in which case, by default, the synthetic object is created with ttl = 1s / grace = 5s / keep = 5s avoid hammering on failing backends (note this is existing behavior).
• Retired the BackendStart log tag - BackendOpen contains all the information from it

APIs / VMODs

• WS_Reserve() is now deprecated and any use should trigger a compiler warning. It is to be replaced by

  • WS_ReserveAll() to reserve all of the remaining workspace

    It will always leave the workspace reserved even if 0 bytes are available, so it must always be followed by a call to WS_Release()

  • WS_ReserveSize() to reserve a fixed amount.

    It will only leave the workspace reserved if the reservation request could be fulfilled.

  We provide a script to help automate this change in the tools/coccinelle subdirectory of the source tree.

• The RST references generated by vmodtool.py have been changed to match better the VCL syntax to avoid overhead where references are used. The new scheme for a vmod called name is:

  • $Function: name.function()
  • $Object constructor: name.object()
  • $Method: xobject.method()

  To illustrate, the old references:

  :ref:`vmod_name.function`
  :ref:`vmod_name.obj`
  :ref:`vmod_name.obj.method`
  

  now are renamed to:

  :ref:`name.function()`
  :ref:`name.obj()`
  :ref:`xobj.method()`
  

  tools/vmod_ref_rename.sh is provided to automate this task

Varnish Cache 6.2.0 (2019-03-15)

• Extend JSON support in the CLI (2783)

• Improve accuracy of statistics (VSC)

• In Error: out of workspace log entries, the workspace name is now reported in lowercase

• Adjust code generator python tools to python 3 and prefer python 3 over python 2 where available

• Added a thread pool watchdog which will restart the worker process if scheduling tasks onto worker threads appears stuck. The new parameter thread_pool_watchdog configures it. (2418)

• Changed ExpKill log tags to emit microsecond-precision timestamps instead of nanoseconds (2792)

• Changed the default of the thread_pool_watchdog parameter to 60 seconds to match the cli_timeout default

• VSB quoted output has been unified to three-digit octal, VSB_QUOTE_ESCHEX has been added to prefer hex over octal quoting

• Retired long deprecated parameters (VIP16). Replacement mapping is: shm_reclen -> vsl_reclen vcl_dir -> vcl_path vmod_dir -> vmod_path

• The width of the columns of the backend.list cli command output is now dynamic.

  For best forward compatibility, we recommend that scripts parse JSON output as obtained using the -j option.

  See release notes for details.

• The format of the backend.list -j (JSON) cli command output has changed.

  See release notes for details.

• The undocumented -v option to the backend.list cli command has been removed

• Changed the formatting of the vcl.list command from:

  status      state/temperature       busy    name    [labelinfo]
  

  to:

  status      state   temperature     busy    name    [<-|->] [info]
  

  Column width is now dynamic.

  Field values remain unchanged except for the label information, see varnish-cli(7) for details.

• The ban facility has been extended by bans access to obj.ttl, obj.age, obj.grace and obj.keep and additional inequality operators.

• Many cache lookup optimizations.

• Display the VCL syntax during a panic.

• Update to the VCL diagrams to include hit-for-miss.

VCL

• Added req.is_hitmiss and req.is_hitpass (2743)

bundled vmods

• Added directors.lookup()

bundled tools

• Improved varnish log client performance (2788)

• For varnishtest -L, also keep VCL C source files

• Add param.reset command to varnishadm

• Add VSL rate limiting (2837)

  This adds rate limiting to varnishncsa and varnishlog.

• Make it possible to change varnishstat update rate. (2741)

C APIs (for vmod and utility authors)

• libvarnish: VRT_VSA_GetPtr renamed to VSA_GetPtr

• Included vtree.h in the distribution for vmods and renamed the red/black tree macros from VRB_* to VRBT_* to disambiguate from the acronym for Varnish Request Body.

  Changed the internal organisation of dynamic PRIVs (PRIV_TASK, PRIV_TOP from a list to a red/black tree) for performance. (2813)

• Vmod developers are advised that anything returned by a vmod function/method is assumed to be immutable. In other words, a vmod must not modify any data which was previously returned.

• Tolerate null IP addresses for ACL matches.

• Added vstrerror() as a safe wrapper for strerror() to avoid a NULL pointer dereference under rare conditions where the latter could return NULL. (2815)

• Varnish-based tools using the VUT interface should now consider using the VUT_Usage() function for consistency

• The name of the event_function callback for VCL events in vmods is now prefixed by $Prefix_` if $Prefix is defined in the .vcc file, or vmod_ by default.

  So, for example, with $Event foo and no $Prefix, the event function will be called vmod_foo and with $Prefix bar it will be called bar_foo.

• In the vmodtool-generated ReStructuredText documentation, anchors have been renamed

  • from obj_class to vmodname.class for constructors and
  • from func_class to vmodname.function for functions and
  • from func_class to vmodname.class.method for methods,

  repsectively. In short, the anchor is now named equal to VCL syntax for constructors and functions and similarly to VCL syntax for methods.

• VRT API has been updated to 9.0

  • HTTP_Copy() was removed, HTTP_Dup() and HTTP_Clone() were added

  • Previously, VCL_BLOB was implemented as struct vmod_priv, which had the following shortcomings:

    • blobs are immutable, but that was not reflected by the priv pointer
    • the existence of a free pointer suggested automatic memory management, which did never and will not exist for blobs.

    The VCL_BLOB type is now implemented as struct vrt_blob, with the blob member replacing the former priv pointer and the free pointer removed.

    A type member was added for lightweight type checking similar to the miniobject magic member, but in contrast to it, type should never be asserted upon.

    VRT_blob() was updated accordingly.

  • req->req_bodybytes was removed. Replacement code snippet:

    AZ(ObjGetU64(req->wrk, req->body_oc, OA_LEN, &u));
    

  • VRT_SetHealth() has been removed and VRT_SetChanged() added. VRT_LookupDirector() (only to be called from CLI contexts) as been added.

    See release notes for details

• vmodtool has been changed significantly to avoid various name clashes. Rather than using literal prefixes/suffixes, vmod authors should now (and might have to for making existing code continue to compile) use the following macros

  • VPFX(name) to prepend the vmod prefix (vmod_ by default)

  • VARGS(name) as the name of a function/method's argument struct, e.g.:

    VCL_VOID vmod_test(VRT_CTX, struct VARGS(test) *args) { ...
    

  • VENUM(name) to access the enum by the name name

Fixed bugs

• Fixed varnishhist display error (2780)
• Fix varnishstat -f in curses mode (interactively, without -1, 2787)
• Handle an out-of-workspace condition in HTTP/2 delivery more gracefully (2589)
• Fixed regression introduced just before 6.1.0 release which caused an unnecessary incompatibility with VSL files written by previous versions. (2790)
• Fix warmup/rampup of the shard director (2823)
• Fix VRT_priv_task for calls from vcl_pipe {} (2820)
• Fix assinging <bool> == <bool> (2809)
• Fix vmod object constructor documentation in the vmodtool.py - generated RST files
• Fix some stats metrics (vsc) which were wrongly marked as _gauge_
• Fix varnishd -I (2782)
• Add error handling for STV_NewObject() (2831)
• Fix VRT_fail for 'if'/'elseif' conditional expressions (2840)

Varnish Cache 6.1.0 (2018-09-17)

• Added -p max_vcl and -p max_vcl_handling for warnings/errors when there are too many undiscarded VCL instances. (2713)
• Content-Length header is not rewritten in response to a HEAD request, allows responses to HEAD requests to be cached independently from GET responses.

VCL

• return(fail("mumble")) can have a string argument that is emitted by VCC as an error message if the VCL load fails due to the return. (2694)
• Improved VCC error messages (2696)
• Fixed obj.hits in vcl_hit (had been always 0) (2746)

bundled tools

• varnishhist: Improved test coverage
• varnishtest: Added haproxy CLI send/expect facility

C APIs (for vmod and utility authors)

• libvarnishapi so version bumped to 2.0.0 (2718)
• For VMOD methods/functions with PRIV_TASK or PRIV_TOP arguments, the struct vrt_priv is allocated on the appropriate workspace. In the out-of-workspace condition, VCL failure is invoked, and the VMOD method/function is not called. (2708)
• Improved support for the VCL STRANDS type, VMOD blob refactored to use STRANDS (2745)

Fixed bugs

• A series of bug fixes related to excessive object accumulation and Transient storage use in the hit-for-miss case (2760, 2754, 2654, 2763)
• A series of fixes related to Python and the vmodtool (2761, 2759, 2742)
• UB in varnishhist (2773)
• Allow to not have randomness in file_id (2436)
• b64.vtc unstable (2753)
• VCL_Poll ctx scope (2749)

Varnish Cache 6.0.1 (2018-08-29)

• Added std.fnmatch() (2737)
• The variable req.grace is back. (2705)
• Importing the same VMOD multiple times is now allowed, if the file_id is identical.

varnishstat

• The counters

  • sess_fail_econnaborted
  • sess_fail_eintr
  • sess_fail_emfile
  • sess_fail_ebadf
  • sess_fail_enomem
  • sess_fail_other

  now break down the detailed reason for session accept failures, the sum of which continues to be counted in sess_fail.

VCL and bundled VMODs

• VMOD unix now supports the getpeerucred(3) case.

bundled tools

• varnishhist: The format of the -P argument has been changed for custom profile definitions to also contain a prefix to match the tag against.
• varnishtest: syslog instances now have to start with a capital S.

Fixed bugs which may influence VCL behavior

• When an object is out of grace but in keep, the client context goes straight to vcl_miss instead of vcl_hit. The documentation has been updated accordingly. (2705)

Fixed bugs

• Several H2 bugs (2285, 2572, 2623, 2624, 2679, 2690, 2693)
• Make large integers work in VCL. (2603)
• Print usage on unknown or missing arguments (2608)
• Assert error in VPX_Send_Proxy() with proxy backends in pipe mode (2613)
• Holddown times for certain backend connection errors (2622)
• Enforce Host requirement for HTTP/1.1 requests (2631)
• Introduction of '-' CLI prefix allowed empty commands to sneak through. (2647)
• VUT apps can be stopped cleanly via vtc process -stop (2649, 2650)
• VUT apps fail gracefully when removing a PID file fails
• varnishd startup log should mention version (2661)
• In curses mode, always filter in the counters necessary for the header lines. (2678)
• Assert error in ban_lurker_getfirst() (2681)
• Missing command entries in varnishadm help menu (2682)
• Handle string literal concatenation correctly (2685)
• varnishtop -1 does not work as documented (2686)
• Handle sigbus like sigsegv (2693)
• Panic on return (retry) of a conditional fetch (2700)
• Wrong turn at cache/cache_backend_probe.c:255: Unknown family (2702, 2726)
• VCL failure causes TASK_PRIV reference on reset workspace (2706)
• Accurate ban statistics except for a few remaining corner cases (2716)
• Assert error in vca_make_session() (2719)
• Assert error in vca_tcp_opt_set() (2722)
• VCL compiling error on parenthesis (2727)
• Assert error in HTC_RxPipeline() (2731)

Varnish Cache 6.0.0 (2018-03-15)

Usage

• Fixed implementation of the max_restarts limit: It used to be one less than the number of allowed restarts, it now is the number of return(restart) calls per request.

• The cli_buffer parameter has been removed

• Added back umem storage for Solaris descendants

• The new storage backend type (stevedore) default now resolves to either umem (where available) or malloc.

• Since varnish 4.1, the thread workspace as configured by workspace_thread was not used as documented, delivery also used the client workspace.

  We are now taking delivery IO vectors from the thread workspace, so the parameter documentation is in sync with reality again.

  Users who need to minimize memory footprint might consider decreasing workspace_client by workspace_thread.

• The new parameter esi_iovs configures the amount of IO vectors used during ESI delivery. It should not be tuned unless advised by a developer.

• Support Unix domain sockets for the -a and -b command-line arguments, and for backend declarations. This requires VCL >= 4.1.

VCL and bundled VMODs

• return (fetch) is no longer allowed in vcl_hit {}, use return (miss) instead. Note that return (fetch) has been deprecated since 4.0.

• Fix behaviour of restarts to how it was originally intended: Restarts now leave all the request properties in place except for req.restarts and req.xid, which need to change by design.

• req.storage, req.hash_ignore_busy and req.hash_always_miss are now accessible from all of the client side subs, not just vcl_recv{}

• obj.storage is now available in vcl_hit{} and vcl_deliver{}.

• Removed beresp.storage_hint for VCL 4.1 (was deprecated since Varnish 5.1)

  For VCL 4.0, compatibility is preserved, but the implementation is changed slightly: beresp.storage_hint is now referring to the same internal data structure as beresp.storage.

  In particular, it was previously possible to set beresp.storage_hint to an invalid storage name and later retrieve it back. Doing so will now yield the last successfully set stevedore or the undefined (NULL) string.

• IP-valued elements of VCL are equivalent to 0.0.0.0:0 when the connection in question was addressed as a UDS. This is implemented with the bogo_ip in vsa.c.

• beresp.backend.ip is retired as of VCL 4.1.

• workspace overflows in std.log() now trigger a VCL failure.

• workspace overflows in std.syslog() are ignored.

• added return(restart) from vcl_recv{}.

• The alg argument of the shard director .reconfigure() method has been removed - the consistent hashing ring is now always generated using the last 32 bits of a SHA256 hash of "ident%d" as with alg=SHA256 or the default.

  We believe that the other algorithms did not yield sufficiently dispersed placement of backends on the consistent hashing ring and thus retire this option without replacement.

  Users of .reconfigure(alg=CRC32) or .reconfigure(alg=RS) be advised that when upgrading and removing the alg argument, consistent hashing values for all backends will change once and only once.

• The alg argument of the shard director .key() method has been removed - it now always hashes its arguments using SHA256 and returns the last 32 bits for use as a shard key.

  Backwards compatibility is provided through vmod blobdigest with the key_blob argument of the shard director .backend() method:

  • for alg=CRC32, replace:

    <dir>.backend(by=KEY, key=<dir>.key(<string>, CRC32))
    

    with:

    <dir>.backend(by=BLOB, key_blob=blobdigest.hash(ICRC32,
      blob.decode(encoded=<string>)))
    

    Note: The vmod blobdigest hash method corresponding to the shard director CRC32 method is called ICRC32

• for alg=RS, replace:

  <dir>.backend(by=KEY, key=<dir>.key(<string>, RS))
  

  with:

  <dir>.backend(by=BLOB, key_blob=blobdigest.hash(RS,
    blob.decode(encoded=<string>)))
  

• The shard director now offers resolution at the time the actual backend connection is made, which is how all other bundled directors work as well: With the resolve=LAZY argument, other shard parameters are saved for later reference and a director object is returned.

  This enables layering the shard director below other directors.

• The shard director now also supports getting other parameters from a parameter set object: Rather than passing the required parameters with each .backend() call, an object can be associated with a shard director defining the parameters. The association can be changed in vcl_backend_fetch() and individual parameters can be overridden in each .backend() call.

  The main use case is to segregate shard parameters from director selection: By associating a parameter object with many directors, the same load balancing decision can easily be applied independent of which set of backends is to be used.

• To support parameter overriding, support for positional arguments of the shard director .backend() method had to be removed. In other words, all parameters to the shard director .backend() method now need to be named.

• Integers in VCL are now 64 bits wide across all platforms (implemented as int64_t C type), but due to implementation specifics of the VCL compiler (VCC), integer literals' precision is limited to that of a VCL real (double C type, roughly 53 bits).

  In effect, larger integers are not represented accurately (they get rounded) and may even have their sign changed or trigger a C compiler warning / error.

• Add VMOD unix.

• Add VMOD proxy.

Logging / statistics

• Turned off PROXY protocol debugging by default, can be enabled with the protocol debug flag.
• added cache_hit_grace statistics counter.
• added n_lru_limited counter.
• The byte counters in ReqAcct now show the numbers reported from the operating system rather than what we anticipated to send. This will give more accurate numbers when e.g. the client hung up early without receiving the entire response. Also these counters now show how many bytes was attributed to the body, including any protocol overhead (ie chunked encoding).