The Optimize transfers have a performance issue. If the travel request return itineraries with many transit legs and transfers(more than 5) and many options for each transfers(we have seen up to 30 options for each pair of transit legs), then the number of permutations grow exponentially. This lead to extensive growth in memory usage. In our case, the instances run out of memory and became super slow - they did not go down. Since the OTP client did not receive a response it kept resending the request. This resulted in more instances becoming "temporary unavailable" and bringing the entire service down. Our auto-scaling is based on CPU usage, but the CPU usage staid below the limit. This again resulted in very few new instances being created and the latency increased. The stalled instance would eventually come back up, but it would take as long as 20-30 minutes, so restarting them worked better.
This issue should be straight forward to fix and we hope to have a fixe ready in a couple of days.
Version of OTP used (exact commit hash or JAR name)
dev-2.x with the OptimizedTransfer feature turned on
The Optimize transfers have a performance issue. If the travel request return itineraries with many transit legs and transfers(more than 5) and many options for each transfers(we have seen up to 30 options for each pair of transit legs), then the number of permutations grow exponentially. This lead to extensive growth in memory usage. In our case, the instances run out of memory and became super slow - they did not go down. Since the OTP client did not receive a response it kept resending the request. This resulted in more instances becoming "temporary unavailable" and bringing the entire service down. Our auto-scaling is based on CPU usage, but the CPU usage staid below the limit. This again resulted in very few new instances being created and the latency increased. The stalled instance would eventually come back up, but it would take as long as 20-30 minutes, so restarting them worked better.
This issue should be straight forward to fix and we hope to have a fixe ready in a couple of days.
Version of OTP used (exact commit hash or JAR name)
dev-2.x with the OptimizedTransfer feature turned on