Using rewrite_table_path iceberg procedure on a backup

[McKMarcBruchner]

Query engine

Spark 3.4.3

Question

Hi Iceberg team,

I was wondering how to best use the rewrite_table_path procedure on a Backup.

My situation is the following:

• I have an S3 bucket on which Iceberg stores the data and metadata files
• My metastore is being stored in a Hive metastore in a Postgres DB on RDS
• I have a backup of that S3 bucket on another S3 bucket in another region, maybe even another account
• I also have a backup of the RDS on the other account
• Let's say my original S3 bucket got corrupted or I can't reach it anymore, so I need to switch to the backup bucket and backup RDS
• Now I wanted to use rewrite_table_path and register_table to recreate the tables so that I can use them

What I gather from the documentation:

• the rewrite_table_path needs to have a registered table to work, because you are specifying the table name in the CALL command
• on the other hand it says that only after I have run rewrite_table_path, I should run register_table with the new metadata.json. Which makes total sense to me.

My problem is now, how can I run rewrite_table_path without registering the table first? In this case, Spark returns me a Couldn't load table, which makes sense, because the table does not exist.

And in case I first register the table, Spark returns another error Path s3a://backup-bucket/test_table/metadata/v1.metadata.json does not start with s3a://original-bucket/test_table/.

I understand how the rewrite_table_path would work if I can run this on my original bucket with the existing table, then move the data and metadata files to a new bucket and run register_table there. But that might not be possible for me if the old bucket got destroyed or corrupted or is otherwise unreachable.

In this blog they state that my approach should work, but I cannot execute 3. Check for File Path Changes Before Recovery because of the problem described above.

I feel that I'm missing something very obvious. Please advise!

[manuzhang]

@McKMarcBruchner If the path in your metadata files starting with s3a://oringal-bucket, you can just copy it back and register the table. If the path starting with s3a://backup-bucket, you can register a table at the backup-bucket, run rewrite_table_path and copy rewritten files to s3a://orignal-bucket.

[McKMarcBruchner]

@manuzhang my problem is the following in terms of disaster recovery:

• I have an original-bucket where our running data lives on
• I have a backup-bucket where we take backups of the original buckets
• Then if disaster strikes, the original-bucket is corrupted, lost, or in any other way unaccessible for me
• That means I only have access to my backup-bucket, that has a snapshot of the data of the original-bucket
• Meaning I have the data and metadata on the backup-bucket, but the metadata files still contain the s3a://oringal-bucket path
• It is not guaranteed that we can ever go back or reuse the original-bucket name, because bucket names in s3 are global
• Therefore, I would need to be able to run rewrite_table_path on the backup-bucket with the metadata files still containing s3a://oringal-bucket without registering it first.
  This blog states that this should be possible, but I don't see it.

[manuzhang]

I don't think it's possible. Without a table, we can't know which files to rewrite. On the other hand, you may manually replace the paths in the metadata files with sed and other tools.

[McKMarcBruchner]

Would it be possible to use the current metadata file for referencing the "table" even though we have no table in the metastore? Because technically, there is no change we are doing in the metastore when running rewrite_table_path, we're only updating the metadata files and the current metadata file should have all the information required. Just like it does when we run register_table.

[manuzhang]

When updating the metadata files, you have to go through the table API. In other words, you need to have a table first in whatever catalog. It's a limitation currently and can be enhanced in the future with relative path.

[funcbro]

I ran into the same issue.
Currently because of the limitation, the rewrite_table_path is kind of useless.

On the other hand, I follow the relative-path topic in Iceberg since quite some time now, which would actually solve the issue in the first place - but sadly there does not seem to be any progress here ;(

• Support relative paths in Table Metadata #1617 (2020)
• How to move Iceberg table from one location to another #3142 (2021)

As I guess relative-path support will still need some time to be fully specified etc - Is there any possibility that the rewrite_table_path util can be improved, so (desaster) recovery use-cases can be executed?

Othewise we will need to implement some hack solution around - which obviously we want to avoid