Queries the actual store to fetch latest snapshot while serving the URL endpoint.

[abdasgupta]

What this PR does / why we need it:
Currently, if someone queries /snapshot/latest endpoint in backup-restore server, latest snapshots are served from whatever is stored in the already running snapshotter. But, ideally, the snapshots should be fetched from the actual backend store and then serve to the end users. This PR fetches the latest snapshots from actual store when an end user executes /snapshot/latest endpoint.

One use case of this PR is e2e tests for compaction. Full snapshots are stored in same storage by both the snapshotter and compactor. Currently, /snapshot/latest endpoint serves the latest snapshots from the details stored in snapshotter. While snapshotter has the information about what latest snapshots are pushed by itself, it does not have the information about the full snapshots pushed by compactor which runs separately. So, in e2e test for compaction, if we want to check the latest snapshots through this endpoint after a compaction job, this endpoint provides stale data about full snapshot as it only has the info about full snapshots pushed by the snapshotter.

Which issue(s) this PR fixes:
Fixes partial gardener/etcd-druid#704

Special notes for your reviewer:

Release note:

The snapshots are fetched from the actual backend store when queried for latest snapshots on `/snapshot/latest` endpoint.

[ishan16696]

But, ideally, the snapshots should be fetched from the actual backend store and then serve to the end users. This PR fetches the latest snapshots from actual store when an end user executes /snapshot/latest endpoint.

but why it should be fetched from snapstore as backup-restore leader already have this information as it is taking the snapshot, so why not use that rather then go to snapstore and list the snapshots ? Have you seen any weird behaviour like when /latest doesn't give latest snapshot information ?

[abdasgupta]

But, ideally, the snapshots should be fetched from the actual backend store and then serve to the end users. This PR fetches the latest snapshots from actual store when an end user executes /snapshot/latest endpoint.

but why it should be fetched from snapstore as backup-restore leader already have this information as it is taking the snapshot, so why not use that rather then go to snapstore and list the snapshots ? Have you seen any weird behaviour like when /latest doesn't give latest snapshot information ?

Yes. I have seen weird behaviour while developing the e2e tests for compaction. Ideally, a URL endpoint should always fetch data from actual backend, not from any cache or something. Full snapshots are stored in same storage by both the snapshotter and compactor. While snapshotter has the information about what latest snapshots are pushed by itself, it does not have the information about the full snapshots pushed by compactor which runs separately. So, in e2e test for compaction, if we want to check the latest snapshots through this endpoint after a compaction job, this endpoint provides stale data about full snapshot as it only have the info about full snapshots pushed by the snapshotter.

[ishan16696]

While snapshotter has the information about what latest snapshots are pushed by itself, it does not have the information about the full snapshots pushed by compactor which runs separately. So, in e2e test for compaction, if we want to check the latest snapshots through this endpoint after a compaction job, this endpoint provides stale data about full snapshot as it only have the info about full snapshots pushed by the snapshotter.

thanks for clarifying, now it make sense. I think it should be mentioned in PR description as it seems important point to me.

[ishan16696]

LGTM!!

[abdasgupta]

I updated the PR description @ishan16696