Add gauge metrics for number of existing backups and restores

[fabito]

Closes #1077
Signed-off-by: fabito fuechi@ciandt.com

[skriss]

So far so good. I'm thinking maybe we call these metrics backup_count and restore_count? @nrb WDYT about names?

[nrb]

I like backup_count and restore_count, but I'm concerned they don't conform to convention.

Reading https://prometheus.io/docs/practices/naming/, I wonder if total_backups or total_restores would be better. That gives us the plural form and the unit (backup/restore) in the name.

[skriss]

Based on that sounds like we should use backups_total / restores_total (similar to http_requests_total).

[nrb]

Good catch. I'm on board with backup_total and restore_total.

[fabito]

I didn't like the names I picked either. I thought about the _count suffix but found it would be more appropriate for counters.

[nrb]

For updating these, I think a a go routine for updating the counters or in the resync functions (as proposed here) would be ideal. This way, they're kept up-to-date without having to worry about changing all locations that a backup/restore could be created or deleted. That's particularly true for restores, since they can be deleted directly from the client side.

[fabito]

Sorry but I still can't tell which option is better/cleaner (haven't studied go routines yet).
If I take the resync functions path. I was thinking on setting the resyncFunc in NewBackupController function:

c.resyncFunc = c.resync
c.resyncPeriod = syncPeriod

And then create the resync function to set the gauge value:

func (c *backupController) resync() {
	backups, err := c.lister.List(labels.Everything())
        c.metrics.SetBackupTotal(len(backups));
}

And finally do the same for restore_total. Does it make sense ?

[skriss]

@fabito I think this looks reasonable as-is -- it's nice and simple, no need to manage our own goroutines for metrics updates.

[skriss]

@carlisia or @nrb PTAL

[skriss]

@fabito if you're interested in continuing to contribute to velero, I'm happy to help find new issues and provide some direction! We've really appreciated your contributions so far. Let me know.

[fabito]

Sure @skriss , point me to another issue and I'll do my best.

[skriss]

@fabito #1410 is a relatively small UX enhancement that would be nice to get into v1.0. It would be really nice if we could audit all of the CLI commands and check for any similar missing validation.

#1136 is another metrics request that seems reasonable to add.