e2e flake: Permissionless subnets fails due to timeout

[marun]

Symptom

Permissionless e2e test times out even with 2m timeout.

Maybe related: #1994

Sample logs

• [FAILED] [124.929 seconds]
[P-Chain] [Permissionless Subnets]
/home/runner/work/avalanchego/avalanchego/tests/e2e/describe.go:25
  [It] subnets operations [xp, permissionless-subnets]
  /home/runner/work/avalanchego/avalanchego/tests/e2e/p/permissionless_subnets.go:30

  Begin Captured GinkgoWriter Output >>
     targeting node NodeID-12xaHGtAuFU3YcGNuXzbFCYHKyFRZFpvx with URI: http://127.0.0.1:40233/
     allocated funded key(s): [PrivateKey-28vKMty9J2XbbMAnGdramcorPasVNYcmJDd3Tngd4i8DcqmA3L]
     initializing a new wallet for node NodeID-12xaHGtAuFU3YcGNuXzbFCYHKyFRZFpvx with URI: http://127.0.0.1:40233/ 
    STEP: retrieving the node ID of a primary network validator 09/07/23 21:17:22.272
    STEP: create a permissioned subnet 09/07/23 21:17:22.273
     issued transaction with ID: nbDujWVUFBUqEJTa7PW2TucjXyPhhcPPN99YnFPh8sVsXEWh3
    STEP: create a custom asset for the permissionless subnet 09/07/23 21:17:27.181
     issued transaction with ID: 2iQeHUVSghtUJJPWUVezm3ufRU3uJNeujZsByg9nU6wVaqkQrc
  << End Captured GinkgoWriter Output

  Expected
      <context.deadlineExceededError>: {}
  to be nil
  In [It] at: /home/runner/work/avalanchego/avalanchego/tests/e2e/p/permissionless_subnets.go:96

Failing jobs

Linked jobs include a testnet-data artifact containing the network data for the run.

• https://github.com/ava-labs/avalanchego/actions/runs/6114790631/job/16597088932#step:5:3473

[marun]

I think the consensus on this is that the switch to building avalanchego for testing with race detection enabled has broken previous expectations of reasonable timeouts. Extending the timeout for most operations to a maximum upper bound has been suggested to avoid timing flakes like this one.

[marun]

Timeouts have been bumped to 2m but the problem persists, suggesting that timing is not the culprit.