Improve checkpoint publication scheduling

[phbnf]

This PR schedules checkpoint publication operations as early as possible, as per `checkpoint_interval and the last publication timestamp.

[AlCutter]

I'm wondering if we can do this with no visible API changes...

Inside each of the publishCheckpoint routines, rather than polling frequently, we could instead do a better job of scheduling attempts so they have more chance to succeed; e.g. when we check when the last CP was published and discover that it's too new, we now know exactly what the interval must be before the next one can be published.

Would something along the lines of the sketch below work?

const retry := 200*time.Milliseconds // whatever
t := time.NewTicker(retry)

for {
   select {
     case <-ctx.Done(): 
       // whatever
     case <-t.C:
   }
   
   checkpointAge := ...
   if !shouldPublish {
     t.Reset(minInterval-checkpointAge)
     continue
   }
   if err := publishCheckpoint(); err != nil {
     t.Reset(retry)
     continue
   }
   t.Reset(minInterval)
}

[phbnf]

Ah yes 100%, that was my original design but I had understood you didn't want do to that, sorry. I'll get back to it.

[AlCutter]

Ah yes 100%, that was my original design but I had understood you didn't want do to that, sorry. I'll get back to it.

Ok, well great!
I don't think we talked about it when we were braining on this together yesterday, but either way - great minds - carry on! :)

[phbnf]

I've done something along those lines, have a look. Instead of fetching the last checkpoint update time in publishCheckpointJob and thus doing extra reads, I've modified the signature of publishCheckpoint. Happy to change the implementation if you'd prefer something else.

I have not implemented a custom retry interval, I think we can just default back to the user provided value, it keeps things simpler.

I also took that as an opportunity to decrease minCheckpointInterval for GCP.

[AlCutter]

Could the 3 instances of this be simplified to t.Reset(max(time.Millisecond, pubInterval - time.Since(publishedAt)))?

[AlCutter]

You could change otel.TraceErr to otel.Trace2 and return (time.Time, error) from the inner func directly to reduce the readability hit from adding the outer if here. (same in other implementations)