ttl-in-cache may not be functioning as expected

[dhairav]

We are using the stable 8.1.x build of ATS, specifically ATS 8.1.1.
The Origin is a standard Nginx deployment, which only sends out Last-Modified in the headers. No Cache-Control headers.
We want to cache different domains for different times like so:

url_regex=example.com/* method=PUT method=POST method=PATCH method=DELETE method=OPTIONS method=GET method=HEAD ttl-in-cache=31622400s
url_regex=example2.com/* method=PUT method=POST method=PATCH method=DELETE method=OPTIONS method=GET method=HEAD ttl-in-cache=3600s
url_regex=example3.com/* method=PUT method=POST method=PATCH method=DELETE method=OPTIONS method=GET method=HEAD ttl-in-cache=200s
url_regex=example4.com/* method=PUT method=POST method=PATCH method=DELETE method=OPTIONS method=GET method=HEAD ttl-in-cache=7d

Our disks are properly mounted (went through other issues where people have faced the problem - so just clarifying beforehand).

CONFIG proxy.config.http.cache.required_headers INT 1

So by the above config - we expect that ATS cache the response from nginx for 31622400 seconds for Site 1 and 3600 seconds for Site 2. But when we use the debug logs to figure out what is actually happening - we realise that it is bypassing the directives in cache.config altogether and using Heuristic lifetimes for objects - typically a refresh hit is sent to the Origin between every 3600 to 86400 seconds since the first iteration. This results in a lot of TCP_REFRESH_HITS - this data is typically what drove us so deep into debugging the same.

Excerpt from the debug logs -

[Sep  6 18:04:57.976] [ET_NET 1] DEBUG: <HttpTransact.cc:7114 (calculate_document_freshness_limit)> (http_match) [1] calculate_document_freshness_limit --- heuristic: date=1662487470, lm=1662473945, time_since_last_modify=13525, f=0.1, freshness_limit = 1352
[Sep  6 18:04:57.976] [ET_NET 1] DEBUG: <HttpTransact.cc:7142 (calculate_document_freshness_limit)> (http_match) [1] calculate_document_freshness_limit --- final freshness_limit = 3600
[Sep  6 18:04:57.977] [ET_NET 1] DEBUG: <HttpTransactHeaders.cc:438 (calculate_document_age)> (http_age) [calculate_document_age] age_value:              0
[Sep  6 18:04:57.977] [ET_NET 1] DEBUG: <HttpTransactHeaders.cc:439 (calculate_document_age)> (http_age) [calculate_document_age] date_value:             1662487470
[Sep  6 18:04:57.977] [ET_NET 1] DEBUG: <HttpTransactHeaders.cc:440 (calculate_document_age)> (http_age) [calculate_document_age] response_time:          1662487470
[Sep  6 18:04:57.978] [ET_NET 1] DEBUG: <HttpTransactHeaders.cc:441 (calculate_document_age)> (http_age) [calculate_document_age] now:                    1662487497
[Sep  6 18:04:57.978] [ET_NET 1] DEBUG: <HttpTransactHeaders.cc:442 (calculate_document_age)> (http_age) [calculate_document_age] now (fixed):            1662487497
[Sep  6 18:04:57.978] [ET_NET 1] DEBUG: <HttpTransactHeaders.cc:443 (calculate_document_age)> (http_age) [calculate_document_age] apparent_age:           0
[Sep  6 18:04:57.978] [ET_NET 1] DEBUG: <HttpTransactHeaders.cc:444 (calculate_document_age)> (http_age) [calculate_document_age] corrected_received_age: 0
[Sep  6 18:04:57.979] [ET_NET 1] DEBUG: <HttpTransactHeaders.cc:445 (calculate_document_age)> (http_age) [calculate_document_age] response_delay:         0
[Sep  6 18:04:57.979] [ET_NET 1] DEBUG: <HttpTransactHeaders.cc:446 (calculate_document_age)> (http_age) [calculate_document_age] corrected_initial_age:  0
[Sep  6 18:04:57.979] [ET_NET 1] DEBUG: <HttpTransactHeaders.cc:447 (calculate_document_age)> (http_age) [calculate_document_age] resident_time:          27
[Sep  6 18:04:57.980] [ET_NET 1] DEBUG: <HttpTransactHeaders.cc:448 (calculate_document_age)> (http_age) [calculate_document_age] current_age:            27
[Sep  6 18:04:57.980] [ET_NET 1] DEBUG: <HttpTransact.cc:7227 (what_is_document_freshness)> (http_match) [1] [what_is_document_freshness] fresh_limit:  3600  current_age: 27
[Sep  6 18:04:57.980] [ET_NET 1] DEBUG: <HttpTransact.cc:7274 (what_is_document_freshness)> (http_match) [1] [..._document_freshness] initial age limit: 3600
[Sep  6 18:04:57.980] [ET_NET 1] DEBUG: <HttpTransact.cc:7330 (what_is_document_freshness)> (http_match) [1] document_freshness --- current_age = 27
[Sep  6 18:04:57.981] [ET_NET 1] DEBUG: <HttpTransact.cc:7331 (what_is_document_freshness)> (http_match) [1] document_freshness --- age_limit   = 3600
[Sep  6 18:04:57.981] [ET_NET 1] DEBUG: <HttpTransact.cc:7332 (what_is_document_freshness)> (http_match) [1] document_freshness --- fresh_limit = 3600
[Sep  6 18:04:57.981] [ET_NET 1] DEBUG: <HttpTransact.cc:7333 (what_is_document_freshness)> (http_seq) [1] document_freshness --- current_age = 27
[Sep  6 18:04:57.982] [ET_NET 1] DEBUG: <HttpTransact.cc:7334 (what_is_document_freshness)> (http_seq) [1] document_freshness --- age_limit   = 3600
[Sep  6 18:04:57.982] [ET_NET 1] DEBUG: <HttpTransact.cc:7335 (what_is_document_freshness)> (http_seq) [1] document_freshness --- fresh_limit = 3600

I believe this behaviour is documented here - https://docs.trafficserver.apache.org/en/8.1.x/admin-guide/files/records.config.en.html#heuristic-expiration

My question is - how do I force ATS to use the cache.config directives when an explicit Cache-Control header is absent in the Origin response.

We would basically like to use a system wherein there are 3 behaviours -

1. Cache for X seconds when Cache-Control headers are absent using cache.config directives
2. Obey Cache-Control headers when they are present
3. Override Cache-Control headers on a per-remap/site basis.

Appreciate your time and inputs on this, thank you.

[dhairav]

Reminder to please take a look at the above issue.

[randall]

Your best bet here would be to not use cache.config and use remap.config + header_rewrite to control cache-control headers.
and doing something like this in header_rewrite:

cond %{READ_RESPONSE_HDR_HOOK} [AND]
cond %{HEADER:Cache-Control} ="" [AND]
cond %{STATUS} >199 [AND] 
cond %{STATUS} <400
    set-header Cache-Control "max-age=300, public"

[bryancall]

If this is still an issue please reopen.