write.metadata.delete-after-commit.enabled not deleting old metadata

[ayush-san]

After reading this mail archive - https://www.mail-archive.com/dev@iceberg.apache.org/msg01416.html, I had turned on write.metadata.delete-after-commit.enabled for my tables in which data is written by a streaming job.

ALTER TABLE hive.db_name.table_name SET TBLPROPERTIES ('write.metadata.delete-after-commit.enabled'='true')

But even after enabling it, S3 size of the table is still very large as it contain all metadata files. Do I need to enable some other property too?

Latest metadata file content

{
  "format-version" : 2,
  "table-uuid" : "7e6356d7-2e6e-40f0-a462-073b4cbd40fc",
  "location" : "S3_LOCATION",
  "last-sequence-number" : 4072,
  "last-updated-ms" : 1627992967421,
  "last-column-id" : 16,
  "schema" : {
    "type" : "struct",
    "fields" : [...]
  },
  "default-spec-id" : 0,
  "partition-specs" : [ {
    "spec-id" : 0,
    "fields" : [ ]
  } ],
  "default-sort-order-id" : 0,
  "sort-orders" : [ {
    "order-id" : 0,
    "fields" : [ ]
  } ],
  "properties" : {
    "engine.hive.enabled" : "true",
    "write.format.default" : "parquet",
    "write.parquet.compression-codec" : "snappy",
    "write.metadata.delete-after-commit.enabled" : "true"
  },
  "current-snapshot-id" : 6552119266625920959,
...

[flyrain]

The property is only used to delete the version files(*.metadata.json), not manifest list files and manifest files. And, it still keep 100(configurable) version files, check

iceberg/core/src/main/java/org/apache/iceberg/BaseMetastoreTableOperations.java

Line 339 in 0480c8c

private void deleteRemovedMetadataFiles(TableMetadata base, TableMetadata metadata) {

I would suggest to check what's inside the dir metadata, since it has 6.3G data. I doubt the version files occupy a big portion of it. Here are some ideas to reduce size of metadata.

1. To expire snapshots and remove the orphan files
2. To compact manifest files

[ayush-san]

@flyrain Earlier I used to run compaction and manifest rewrite daily. But we started getting duplicate rows due to this issue #2308

But I see that now it's been fixed, so I will wait for the new release and start using compaction and expire snapshot action again

[ayush-san]

@flyrain Running expire-snapshot action will trigger a much bigger issue and will cause the job to fail. Issue - #2603

[github-actions]

This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.

[github-actions]

This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'