Implement "total" on sync progress report

[newswangerd]

Is your feature request related to a problem? Please describe.
The progress report on the sync task has a "total" field that's always null. It would be nice if pulp_ansible could display this value so the user knows how much progress has been made.

Describe the solution you'd like

    "progress_reports": [
        {
            "message": "no_change: Checking if remote changed since last sync.",
            "code": "sync.no_change",
            "state": "completed",
            "total": null,                                 <------- Put the total count here
            "done": 0,
            "suffix": null
        },
        {
            "message": "Downloading Artifacts",
            "code": "sync.downloading.artifacts",
            "state": "running",
            "total": null,                                 <------- Put the total count here
            "done": 0,
            "suffix": null
        },
        {
            "message": "Associating Content",
            "code": "associating.content",
            "state": "running",
            "total": null,                                 <------- Put the total count here
            "done": 0,
            "suffix": null
        },
        {
            "message": "Parsing CollectionVersion Metadata",
            "code": "sync.parsing.metadata",
            "state": "running",
            "total": null,                                 <------- Put the total count here
            "done": 868,
            "suffix": null
        },
        {
            "message": "Downloading Artifacts",
            "code": "sync.downloading.artifacts",
            "state": "running",
            "total": null,                                 <------- Put the total count here
            "done": 267,
            "suffix": null
        }
    ],

[newswangerd]

Also, dumb question, why are there two Downloading Artifacts fields with different values?

[mdellweg]

I don't know much about the pulp_ansible sync pipeline details... So this is a general Pulp answer:
The sync pipeline is organized in stages that run in parallel, so downloading starts before all the metadata is digested. So it's generally hard to know the total numbers upfront. OTOH having a running total instead is more confusing. Maybe it's possible to inject the total number once the first stage is finished. But due to backpressure, this may be very late in the process anyway.
Are there two artifact download stages?

[gerrod3]

The two Downloading Artifacts is interesting, not sure what is going on there.

For v3 galaxy syncs if they are using the unpaginated endpoint we know the total number before filtering or dependencies are added. So depending on the sync settings used this number could be way off from the actual number synced. We can always update this total as we go along through the sync if you want to use it as a progress bar.

[newswangerd]

I wouldn't mind having a a running total. Having some indication of how much is left to do is better than none. I also wouldn't mind having an accurate count for at least one of the steps. Just being able to show a progress bar for "downloading artifacts" would be a big help

[newswangerd]

Having some notion that progress is being made is a really important part of user design: https://99percentinvisible.org/episode/wait-wait-tell-me/