Scan all status is hung after migration from 1.8 to 1.10

[danfengliu]

Test Steps:
1. Deploy harbor 1.8.5;
2. Set Scan all shedule to 12 12 12 * * *;
3. Wait more than 12 minutes;
4. Update harbor to 1.10;
5. View page of "Interrogation Services";
Result:
Page is in scan waiting status for long time.

[danfengliu]

[steven-zou]

en?

This is very weird. Total is 0 but ongoing is true.

@AllForNothing which property you're using to disable the button?

[AllForNothing]

on

en?

This is very weird. Total is 0 but ongoing is true.

@AllForNothing which property you're using to disable the button?

ongoing

[reasonerjt]

Let's be clear that this issue is because the Harbor is shutdown while the job is running and the interrupted job is blocked after Harbor is started.

So it does not only impact upgrading.

[danfengliu]

@reasonerjt Yes, it's upgrade regardless.

[steven-zou]

I think the issue can be generalized to the case that, if there is a Redis restart when lots of jobs are in progress, there might be cause job loss issues. Once the jobs could not be recovered, the status data in the database will be out of sync and look like hung there.

[steven-zou]

@stuclem @xaleeks

Could we add release notes for such known issues?

If there are jobs are still running in the job service, and a Redis restart happened with any reasons (power outage, crash etc.), job loss issues may happen. That would cause status out of sync issues that appeared in the scan, replication or GC scenarios. As result, some status will be like hung there, no change anymore.

[stuclem]

@steven-zou yes of course. Added the kind/note label.

[renmaosheng]

@steven-zou we also need to mention the workaround when this happens.

[michmike]

what's the workaround and is it something that happens automatically or someone has to manually execute it?

[michmike]

does this "hung" issue happen 100% on each upgrade?

[steven-zou]

@michmike

No. It will happen only and only if there are jobs are running and then Redis restart occurs for any reason. Most of the time, when users doing their upgrading, there will not be jobs running.

I'll continue to do investigate this issue. The possible root cause is data loss happens after Redis restarts.

Current plan is we can deliver a fix in the patch release 1.10.X.

[steven-zou]

The workaround approaches will be noted in the FAQ list. By manually triggering API calls.

[michmike]

comments after discussion:
-This is not a new issue
-data loss could be: in progress jobs in the queue. jobs could be replication, scanning, GC
....However, replication will be eventually consistent on the next replication
....scanning is safe since next scan will work. for new images, policy will prevent them from being pulled until they are scanned

i think we are fine to release with this ticket and fix it in the next patch release. this is not data loss, because Harbor is eventually consistent based on user intent. A redis cache restart (for whatever reasons) will chase the same effects and Harbor will be eventually consistent.

[stuclem]

Added to the RNs, so removing kind/note: https://github.com/goharbor/harbor/releases/tag/v1.10.0#known-issues

• Scan status freezes after migration from 1.8 to 1.10 #9963
  If there are jobs running in the job service, and a Redis restart happens, for example because of a power outage or crash, status out of sync issues might appear in scan, replication or garbage collection tasks. Consequently, some task statuses freeze in their current state, and do not update themselves. The cause is known and this will be fixed in a patch.

  Workaround: Run the task again. Replication and scanning function correctly on the next run. No data is lost.

[steven-zou]

It may be related to this issue in the upstream project : gocraft/work#146

[steven-zou]

The code work is done, pending for PR review.

Move to next SP 78