backupccl: fix bug where restore would try to start rolled-back jobs

[pbardea]

Previously, if the transaction which publishes the tables ingested
during a restore is restated, we accumulate jobs which no longer exist
in a slice to start. The work function run inside the transaction should
be idempotent and only expose jobs that have actually been created.

Release note (bug fix): Fix an error that may occur at the end of a
restoration of a backup that had ongoing schema change jobs.

[cockroach-teamcity]

This change is 

[pbardea]

This diff was tested by applying the following patch:

diff --git a/pkg/ccl/backupccl/restore_job.go b/pkg/ccl/backupccl/restore_job.go
index 76bf0f2bd2..12afb76897 100644
--- a/pkg/ccl/backupccl/restore_job.go
+++ b/pkg/ccl/backupccl/restore_job.go
@@ -13,6 +13,7 @@ import (
        "context"
        "fmt"
        "math"
+       "math/rand"
        "runtime"
        "sync/atomic"
        "time"
@@ -1133,6 +1134,11 @@ func (r *restoreResumer) publishTables(ctx context.Context) error {
                        return errors.Wrap(err, "updating job details after publishing tables")
                }

+               if rand.Float64() < 0.9 {
+                       return txn.GenerateForcedRetryableError(ctx, "boom")
+               }
+
                return nil
        })
        if err != nil {

and stressing TestRestoreMidSchemaChange before and after the fix (which previously failed instantly). I would like to introduce a more general way to inject txn retries during testing, possibly something like #56074, but for the sake of backporting I think that should be addressed separately.

[pbardea]

TFTR!
bors r=miretskiy

[craig]

Build failed:

• GitHub CI (Cockroach)