chore: big refactor of TestHttpFetch, add basic aggregateevent testing

[rvagg]

Builds on #345 but is separate because this is a major refactor of TestHttpFetch, mainly just moving things around to make it slightly more readable.

Also adds some basic aggregateevent testing into the integration test so we can start to assert it reports things that we expect.

[codecov-commenter]

Codecov Report

Merging #420 (daf4ef7) into feat/add-bitswap-tracking (2dbd2ce) will increase coverage by 0.21%.
The diff coverage is 100.00%.

Additional details and impacted files

@@                      Coverage Diff                      @@
##           feat/add-bitswap-tracking     #420      +/-   ##
=============================================================
+ Coverage                      76.66%   76.87%   +0.21%     
=============================================================
  Files                             86       86              
  Lines                           6325     6332       +7     
=============================================================
+ Hits                            4849     4868      +19     
+ Misses                          1232     1224       -8     
+ Partials                         244      240       -4     

Files Changed	Coverage Δ
...g/aggregateeventrecorder/aggregateeventrecorder.go	90.62% <100.00%> (+0.04%)	⬆️
pkg/server/http/ipfs.go	90.59% <100.00%> (+1.28%)	⬆️
pkg/server/http/servertimingssubscriber.go	100.00% <100.00%> (ø)

... and 8 files with indirect coverage changes

[rvagg]

curious, bitswap tests now failing on windows with long runs (~30s), they really shouldn't run that long regardless

[rvagg]

Comparing Windows test run times on this branch:

    --- PASS: TestHttpFetch/graphsync_large_sharded_file (0.26s)
    --- PASS: TestHttpFetch/bitswap_large_sharded_file (0.43s)
    --- PASS: TestHttpFetch/http_large_sharded_file (2.30s)
    --- PASS: TestHttpFetch/graphsync_large_directory (10.72s)
    --- FAIL: TestHttpFetch/bitswap_large_directory (31.62s)
    --- PASS: TestHttpFetch/http_large_directory (16.49s)
    --- PASS: TestHttpFetch/graphsync_large_sharded_directory (7.66s)
    --- FAIL: TestHttpFetch/bitswap_large_sharded_directory (33.04s)
    --- PASS: TestHttpFetch/http_large_sharded_directory (19.43s)
    --- PASS: TestHttpFetch/graphsync_max_block_limit (2.62s)
    --- PASS: TestHttpFetch/graphsync_max_block_limit_in_request (2.44s)
    --- PASS: TestHttpFetch/bitswap_max_block_limit (7.00s)
    --- PASS: TestHttpFetch/http_max_block_limit (5.67s)
    --- FAIL: TestHttpFetch/bitswap_block_timeout_from_missing_block (32.47s)

To last main test run:

    --- PASS: TestHttpFetch/graphsync_large_sharded_file (0.66s)
    --- PASS: TestHttpFetch/bitswap_large_sharded_file (0.30s)
    --- PASS: TestHttpFetch/http_large_sharded_file (0.70s)
    --- PASS: TestHttpFetch/graphsync_large_directory (0.95s)
    --- PASS: TestHttpFetch/bitswap_large_directory (1.41s)
    --- PASS: TestHttpFetch/http_large_directory (0.95s)
    --- PASS: TestHttpFetch/graphsync_large_sharded_directory (0.86s)
    --- PASS: TestHttpFetch/bitswap_large_sharded_directory (1.14s)
    --- PASS: TestHttpFetch/http_large_sharded_directory (0.94s)
    --- PASS: TestHttpFetch/graphsync_max_block_limit (0.25s)
    --- PASS: TestHttpFetch/graphsync_max_block_limit_in_request (0.28s)
    --- PASS: TestHttpFetch/bitswap_max_block_limit (0.14s)
    --- PASS: TestHttpFetch/http_max_block_limit (0.11s)
    --- PASS: TestHttpFetch/bitswap_block_timeout_from_missing_block (1.13s)

Linux runs are obviously faster but still show notable difference:

PASS: TestHttpFetch/graphsync_large_sharded_file (0.30s)
    --- PASS: TestHttpFetch/bitswap_large_sharded_file (0.41s)
    --- PASS: TestHttpFetch/http_large_sharded_file (0.57s)
    --- PASS: TestHttpFetch/graphsync_large_directory (2.32s)
    --- PASS: TestHttpFetch/bitswap_large_directory (5.17s)
    --- PASS: TestHttpFetch/http_large_directory (1.21s)
    --- PASS: TestHttpFetch/graphsync_large_sharded_directory (1.67s)
    --- PASS: TestHttpFetch/bitswap_large_sharded_directory (4.84s)
    --- PASS: TestHttpFetch/http_large_sharded_directory (1.29s)
    --- PASS: TestHttpFetch/graphsync_max_block_limit (0.45s)
    --- PASS: TestHttpFetch/graphsync_max_block_limit_in_request (0.32s)
    --- PASS: TestHttpFetch/bitswap_max_block_limit (0.38s)
    --- PASS: TestHttpFetch/http_max_block_limit (0.52s)
    --- PASS: TestHttpFetch/bitswap_block_timeout_from_missing_block (1.66s)

 (0.21s)
    --- PASS: TestHttpFetch/bitswap_large_sharded_file (0.11s)
    --- PASS: TestHttpFetch/http_large_sharded_file (0.15s)
    --- PASS: TestHttpFetch/graphsync_large_directory (0.52s)
    --- PASS: TestHttpFetch/bitswap_large_directory (0.45s)
    --- PASS: TestHttpFetch/http_large_directory (0.40s)
    --- PASS: TestHttpFetch/graphsync_large_sharded_directory (0.48s)
    --- PASS: TestHttpFetch/bitswap_large_sharded_directory (0.46s)
    --- PASS: TestHttpFetch/http_large_sharded_directory (0.38s)
    --- PASS: TestHttpFetch/graphsync_max_block_limit (0.17s)
    --- PASS: TestHttpFetch/graphsync_max_block_limit_in_request (0.17s)
    --- PASS: TestHttpFetch/bitswap_max_block_limit (0.07s)
    --- PASS: TestHttpFetch/http_max_block_limit (0.07s)
    --- PASS: TestHttpFetch/bitswap_block_timeout_from_missing_block (1.05s)

Will revisit tomorrow.

[rvagg]

running a profile just on TestHttpFetch I see this in the top 10:

Showing nodes accounting for 39.29s, 78.99% of 49.74s total
Dropped 1012 nodes (cum <= 0.25s)
Showing top 10 nodes out of 190
      flat  flat%   sum%        cum   cum%
    14.51s 29.17% 29.17%     32.03s 64.39%  runtime.selectgo
     5.70s 11.46% 40.63%      5.71s 11.48%  runtime.unlock2
     5.60s 11.26% 51.89%      5.73s 11.52%  runtime.lock2
     2.53s  5.09% 56.98%      8.22s 16.53%  runtime.sellock
     2.51s  5.05% 62.02%      2.51s  5.05%  runtime.memmove
     2.29s  4.60% 66.63%     35.32s 71.01%  github.com/filecoin-project/lassie/pkg/aggregateeventrecorder.(*aggregateEventRecorder).ingestEvents
     1.80s  3.62% 70.25%      7.52s 15.12%  runtime.selunlock
     1.52s  3.06% 73.30%      1.52s  3.06%  crypto/sha256.block
     1.43s  2.87% 76.18%      1.43s  2.87%  runtime/internal/syscall.Syscall6
     1.40s  2.81% 78.99%      1.40s  2.81%  runtime.memclrNoHeapPointers

so just wiring up the aggregateeventrecorder is a huge problem; this is likely going to be an issue in production for Saturn.

[rvagg]

14f244d - missing return in the ingestEvents loop when context cancelled, so we'd have a whole lot of these running in the background just looping endlessly; hence the slow tests.