[Feature Request]: [IcebergIO] Fetch a fresh spec periodically in hash distribution

[ahmedabu98]

What would you like to happen?

In AssignDestinationsAndPartitions, we just fetch the spec once per VM and store the generated PartitionKey.

The spec may change at any time though. To avoid working with a stale spec, we should periodically fetch it from the table

Issue Priority

Priority: 2 (default / most feature requests should be filed as P2)

Issue Components

• Component: Python SDK
• Component: Java SDK
• Component: Go SDK
• Component: Typescript SDK
• Component: IO connector
• Component: Beam YAML
• Component: Beam examples
• Component: Beam playground
• Component: Beam katas
• Component: Website
• Component: Infrastructure
• Component: Spark Runner
• Component: Flink Runner
• Component: Prism Runner
• Component: Twister2 Runner
• Component: Hazelcast Jet Runner
• Component: Google Cloud Dataflow Runner