0.10.1 (2024-02-05)

Features

• Add support of Incremental Strategies for Kafka connection:

  reader = DBReader(
      connection=Kafka(...),
      source="topic_name",
      hwm=AutoDetectHWM(name="some_hwm_name", expression="offset"),
  )
  
  with IncrementalStrategy():
      df = reader.run()

  This lets you resume reading data from a Kafka topic starting at the last committed offset from your previous run. (#202)

• Add has_data, raise_if_no_data methods to DBReader class. (#203)

• Updade VMware Greenplum connector from 2.1.4 to 2.3.0. This implies:

  • Greenplum 7.x support
  • Kubernetes support
  • New read option gpdb.matchDistributionPolicy which allows to match each Spark executor with specific Greenplum segment, avoiding redundant data transfer between Greenplum segments
  • Allows overriding Greenplum optimizer parameters in read/write operations (#208)

• Greenplum.get_packages() method now accepts optional arg package_version which allows to override version of Greenplum connector package. (#208)