Prod 325 databricks rollovers

[rmoneys]

https://synccomputing.atlassian.net/browse/PROD-325

Start with spark_log_parser/__main__.py to see how the new event log module is used.

[gorskysd]

Nice. The compression handling here is really clear, I like the design pattern you put in. Also the robust use of Path was nice, there are a lot of pieces I saw here that I wasn't familiar with.

I made some minor suggestions for change, but nothing critical IMO, so submitting my approval.

[gorskysd]

One last minute change, sorry!

Corner case to consider where there's only one rollover log and it's not the 0th -- in this case I believe the rollover ID check is bypassed. This is a common case when client's don't know to expect more than one eventlog. See example log in https://synccomputing.atlassian.net/browse/PROD-399?focusedCommentId=11208

[rmoneys]

Phew!
Rolled in some good feedback since e5f1974 including,

1. Added support for s3 directories
2. Handled missing first db rollover log corner case
3. Removed pydantic model since it wasn't doing much besides getting in the way of injecting an s3 client to make testing easier
4. Added more tests
5. Misc refactor for more robust code 🤞

[sonarqubecloud]

Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
10 Code Smells

No Coverage information
0.0% Duplication

[gorskysd]

Nice, this looks great, Scott.

I'm learning a lot about Path and url handling here that's super useful.