Auto-scaled workers repeatedly perform car linking

[abyrd]

I'm running a regional analysis on staging of the Basel region, using a car-wait modification. It automatically scaled up to 100 workers. The original worker on which I performed single point requests is working fine, handling over 200 tasks per minute. All the other auto-scaled workers are very slow, mostly at about 5 tasks per minute. Inspection of their logs reveals that they are continuously re-performing the car linkage. The car linkage is apparently being sought on a different street layer instance for every task, and these linkages are being evicted.

Filtering one worker's log for "CAR" we see:

10:20:40
1609204 [pool-2-thread-3] INFO com.conveyal.r5.streets.LinkedPointSet - The base linkage is for street mode CAR
10:20:40
1609204 [pool-2-thread-4] INFO com.conveyal.r5.streets.LinkedPointSet - The base linkage is for street mode CAR
10:20:40
1609204 [pool-2-thread-2] INFO com.conveyal.r5.streets.LinkedPointSet - The base linkage is for street mode CAR
10:20:40
1609204 [pool-2-thread-1] WARN com.conveyal.r5.analyst.PointSet - Linkage cache evicted Tuple2[com.conveyal.r5.streets.StreetLayer@621c98d6, CAR], cause: SIZE
10:20:40
1609204 [pool-2-thread-1] INFO com.conveyal.r5.streets.LinkedPointSet - The base linkage is for street mode CAR
10:20:40
1609239 [pool-2-thread-2] WARN com.conveyal.r5.analyst.PointSet - Linkage cache evicted Tuple2[com.conveyal.r5.streets.StreetLayer@51b152fa, CAR], cause: SIZE
10:20:40
1609287 [pool-2-thread-3] WARN com.conveyal.r5.analyst.PointSet - Linkage cache evicted Tuple2[com.conveyal.r5.streets.StreetLayer@401cd0d6, CAR], cause: SIZE
10:20:40
1609320 [pool-2-thread-4] WARN com.conveyal.r5.analyst.PointSet - Linkage cache evicted Tuple2[com.conveyal.r5.streets.StreetLayer@53aa1427, CAR], cause: SIZE
10:20:40
1609322 [pool-2-thread-1] WARN com.conveyal.r5.analyst.PointSet - Linkage cache evicted Tuple2[com.conveyal.r5.streets.StreetLayer@27aff22a, CAR], cause: SIZE
10:20:41
1610158 [pool-2-thread-2] INFO com.conveyal.r5.analyst.PointSet - Seeking linkage for (com.conveyal.r5.streets.StreetLayer@e60801d, CAR) in cache...
10:20:41
1610158 [pool-2-thread-2] INFO com.conveyal.r5.analyst.PointSet - Linkage for (com.conveyal.r5.streets.StreetLayer@e60801d, CAR) was not found in cache, building it now.
10:20:41
1610158 [pool-2-thread-2] INFO com.conveyal.r5.analyst.PointSet - Seeking linkage for (com.conveyal.r5.streets.StreetLayer@401cd0d6, CAR) in cache...
10:20:41
1610158 [pool-2-thread-2] INFO com.conveyal.r5.analyst.PointSet - Linkage for (com.conveyal.r5.streets.StreetLayer@401cd0d6, CAR) was not found in cache, building it now.
10:20:41
1610810 [pool-2-thread-1] INFO com.conveyal.r5.analyst.PointSet - Seeking linkage for (com.conveyal.r5.streets.StreetLayer@774927b7, CAR) in cache...
10:20:41
1610810 [pool-2-thread-1] INFO com.conveyal.r5.analyst.PointSet - Linkage for (com.conveyal.r5.streets.StreetLayer@774927b7, CAR) was not found in cache, building it now.
10:20:41
1610810 [pool-2-thread-1] INFO com.conveyal.r5.analyst.PointSet - Seeking linkage for (com.conveyal.r5.streets.StreetLayer@401cd0d6, CAR) in cache...

[abyrd]

@ansoncfit pointed out this commented out code in LinkedPointSet that seems very relevant:

// This has been commented out because this was evaluating to true frequently on car searches
	// Perhaps the effect of identity equality comparisons and the fact that both base layer and new linkage are coming from a cache?
	//        if (baseLinkage != null && (
	//                baseLinkage.pointSet != pointSet ||
	//                baseLinkage.streetLayer != streetLayer.baseStreetLayer ||
	//                baseLinkage.streetMode != streetMode)) {
	//            LOG.error("Cannot reuse linkage with mismatched characteristics. THIS IS A BUG.");
	//            // Relink everything as if no base linkage was supplied.
	//            baseLinkage = null;
	//        }

This is being uncommented in PR #463 to reinstate this sanity check.

[abyrd]

After running a single point analysis with CAR+WALK over a whole region, I changed the geographic extents of analysis. In single point analysis this changes the destination grid extents. I would expect relinking to be instantaneous since it's just using a subgrid of the existing full-region CAR grid.

However, it relinks the CAR apparently because it doesn't have a linkage in cache for the baseline network. I would expect that baseline linkage to have been saved when the full-region single point analysis was done.

The original issue reported in this ticket could be related to #514. Regional workers were not applying any scenario (due to #518). This causes it to seek linkages for the true baseline (no scenario) rather than an empty baseline scenario with its own ID. If the CAR linkage for the baseline street network was never saved into the cache, it would be built for every single request.

[ansoncfit]

Discussing with @abyrd, we think the main bug causing this behavior was addressed in #508. But repeated linking might still occur when car linkages are evicted from the linkageCache (which can easily happen given a combination of 3 access modes, two scenarios, and LINKAGE_CACHE_SIZE = 6)

[abyrd]

I think we can close this ticket once the fix to #508 is merged into dev, and continue working on linkage problems under #521.