Add Cache-Control headers on AP endpoints

[thebaer]

This enables future use of Varnish in front of WriteFreely (see T693 for details).

Includes:

• ActivityPub Collection fetching via canonical URL
• ActivityPub Collection fetching via API
• ActivityPub Post fetching via canonical URL
• ActivityPub Post fetching via API

(By "ActivityPub fetching" I mean requests with the header Accept: application/activity+json)

This closes T693.

[ghost]

I'm not sure if you did any benchmarks but I did 😁 on the collection endpoint domain.com/collection

Without Varnish:

Transactions:		        1020 hits
Availability:		      100.00 %
Elapsed time:		        2.14 secs
Data transferred:	        0.71 MB
Response time:		        0.47 secs
Transaction rate:	      476.64 trans/sec
Throughput:		        0.33 MB/sec
Concurrency:		      222.43
Successful transactions:        1020
Failed transactions:	           0
Longest transaction:	        0.98
Shortest transaction:	        0.16

With Varnish:

Transactions:		        1533 hits
Availability:		      100.00 %
Elapsed time:		        3.36 secs
Data transferred:	        0.75 MB
Response time:		        0.41 secs
Transaction rate:	      456.25 trans/sec
Throughput:		        0.22 MB/sec
Concurrency:		      188.11
Successful transactions:        1533
Failed transactions:	           0
Longest transaction:	        0.71
Shortest transaction:	        0.15

And with Varnish in the middle but before the pages are cached the transaction times were:

Longest transaction:	        0.88
Shortest transaction:	        0.33

That's just with two iterations of 255 concurrent users.

[ghost]

Everything looks good and the cache control header is working as expected on the specified end points.

I also tested response times again using wrk, matt is still just Nginx and rob is with Varnish:

~/c/wrk (master|✔) $ ./wrk -t8 -c1000 -H "Accept: application/activity+json" https://matt.writefreely.dev
Running 10s test @ https://matt.writefreely.dev
  8 threads and 1000 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   475.32ms  187.24ms   1.98s    81.42%
    Req/Sec   109.66     50.04   272.00     68.52%
  7587 requests in 10.08s, 10.51MB read
  Socket errors: connect 208, read 3752, write 0, timeout 21
Requests/sec:    752.60
Transfer/sec:      1.04MB
~/c/wrk (master|✔) $ ./wrk -t8 -c1000 -H "Accept: application/activity+json" https://rob.writefreely.dev/log
Running 10s test @ https://rob.writefreely.dev/log
  8 threads and 1000 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   268.45ms  169.69ms   1.95s    90.81%
    Req/Sec   212.60     92.59   630.00     70.68%
  15536 requests in 10.09s, 3.40MB read
  Socket errors: connect 159, read 3170, write 0, timeout 66
Requests/sec:   1539.76
Transfer/sec:    345.50KB

This one supports more concurrent connections, set at 1000 here.

[thebaer]

Nice! Thanks for the thorough testing / benchmarking -- I hadn't done that 😄. And to be sure, this only introduces caching on ActivityPub endpoints where we're doing GET requests?

[ghost]

Yep, the other requests include the Via: Varnish header but no age or cache control. Try curl -IL rob.writefreely.dev to see a non cached page

[ghost]

Actually good call @thebaer, it seems to be caching the post fetch endpoint even without the header for activity json.

[thebaer]

Gotcha. So I think this might be a varnish thing, where we include the Accept header in the hash.

[ghost]

Got it, I just added a case in the subroutine for receive sub vcl_recv. When the request does not contain the accept header for activity json it passes the request direct to the backend. With the header you process through caching. To see it in action, including the cached version aging with non cache requests between, alternate the following a few times:

curl -IL rob.writefreely.dev/log/hello-hubzilla -H "Accept: application/activity+json"

curl -IL rob.writefreely.dev/log/hello-hubzilla

[thebaer]

Thanks, let's continue that discussion on writefreely/documentation#7 -- I think there are some other things we might want to tweak. For now, I'll merge this in.