SparkMicroBatchStream doesn't respect `start-snapshot-id` or `end-snapshot-id`

[kbendick]

Feature Request / Improvement

Currently, the SparkReadConf can be used with start-snapshot-id and end-snapshot-id options to read just a specific chunk of the table.

These options are not respected when SparkMicroBatchStream is initialized, i.e. using the following will silently ignore these options.

 val df =  spark
      .readStream
      .format("iceberg")
      .option("start-snapshot-id", startId)
      .option("end-snapshot-id", endId)
      .table("rauls_table")

We should implement support for more than jsut timestamp based incremental reads in SparkMicrobatchStream.

This was reported on Slack and I'm opening an issue for it. Please feel free to work on this if you have time 👍

Query engine

Spark

[github-actions]

This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.

[github-actions]

This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'