Allow batch size of artifacts processed during sync to be configured

[balasankarc]

closes #7024

Instead of hardcoding a value of 200, which can cause the storage requirements to increase drastically, allow batch size of artifacts processed during sync to be configured by users.

[dkliban]

@balasankarc Thank you for the contribution. The CI failures are not related to your changes. We will accept this change once we have our CI working.

In the future it would better if this setting was available on each remote. Setting it globally to a low number will cause significantly more load on the database.

[balasankarc]

@balasankarc Thank you for the contribution. The CI failures are not related to your changes. We will accept this change once we have our CI working.

In the future it would better if this setting was available on each remote. Setting it globally to a low number will cause significantly more load on the database.

@dkliban I'll work on the setting being per-remote as a different PR, as next iteration. So this one can solve the immediate problem and unblock us, and the next one enhances it to be better from a performance PoV.

[dralley]

FYI, CI should be fixed once you rebase onto the latest state of the main branch

[balasankarc]

@dralley Done.

[balasankarc]

@ggainey Is there a regular minor version release coming out soon, that may include this fix? If not, would it be possible to have a patch version out with this fix?

[ggainey]

@ggainey Is there a regular minor version release coming out soon, that may include this fix? If not, would it be possible to have a patch version out with this fix?

We generally do new Y-releases as needed, on Tuesdays. As it stands, I not sure we can backport this - what happens to existing installs that don't have the new setting, but install the new code? @dkliban wdyt?

[balasankarc]

@ggainey Is there a regular minor version release coming out soon, that may include this fix? If not, would it be possible to have a patch version out with this fix?

We generally do new Y-releases as needed, on Tuesdays. As it stands, I not sure we can backport this - what happens to existing installs that don't have the new setting, but install the new code? @dkliban wdyt?

@ggainey That's fine. Is there a -Y release planned this week, which can include this?

/cc @bmbouter

[ggainey]

Yessir, "having a .feature available" is what triggers a new-Y on Tuesdays :)

[balasankarc]

Awesome. TIL. Thanks.