Update metadata cleanup to delete only expired files non needed to time travel for non expired versions

[JassAbidi]

This PR updates the metadata cleanup behavior to delete only expired files non needed to time travel to non expired version.
Current behavior:
A Delta log is deleted when it's older than the retention period and there's a checkpoint file after it. The problem with such logic is that it can lead to situations where time travel is not available for non expired versions. this issue shows an example of that.
Proposed behavior:
when an expired file is needed to time travel for a non expired version, we keep it in the transaction log. We only cleanup expired logs when they are no longer needed to time travel to any non expired version.

• closes issues Checkpoint is preventing application from accessing previous delta table versions newer than logRetentionDuration #255 and I receive the error “Cannot time travel Delta table to version X” whereas I can see the version X when looking at the history #616

[scottsand-db]

Hi @JassAbidi. Thanks for this PR. As of now we do not consider this issue to be a bug, though we may consider in the future revisiting this and implementing something similar to your solution here.

In Delta Lake 1.1 docs here, we've added a note explaining that this can happen and that increasing delta.logRetentionDuration can help avoid these situations.

Going to close this PR for now, and if we revisit this issue later we can take another look.