[Feature] Run Storage Template Migration concurrently

[n2iw]

I have searched the existing feature requests, both open and closed, to make sure this is not a duplicate request.

• Yes

The feature

Currently, manually run Storage Template Migration only creates one job. There are at least three issues with it:

1. It runs painfully slow in large libraries (I have 900K assets), because it cannot run concurrently
2. There is no way to know migration progress.
3. There is no way to pause and resume, which makes it even more painful

I propose separating the huge job into small jobs, one per each asset, just like Extract Metadata or Generate Thumbnails. So that it can run concurrently, and can pause and resume too.

Platform

• Server
• Web
• Mobile

[alextran1502]

We used to have it run concurrently, but due to the data-race nature of this type of operation, the concurrency level of this job has been removed

[n2iw]

Do you mind elaborate on details of "data-race"? Thanks!

[bo0tzz]

Two assets might be attempted to move into the same path

[n2iw]

Got it! I wonder, if with some luck, I could fix the same target location issue, and used a batched job instead of one job per asset, would the development team be open to accepting a PR for this? Or it's a hard no?

[bo0tzz]

In addition, queueing this job for each asset independently would slow things down pretty substantially.

[n2iw]

Isn't that the same situation as Extract Metadata and Generate Thumbnail?

[bo0tzz]

No, those do a lot more work.

[n2iw]

I see, so I assume the performance concern is related to the overhead of adding all assets individually to the queue? Make sense to me, thanks for the prompt reply!