Added the SymlinkTextInputFormat manifest generation for Presto/Athena support

[tdas]

This PR is the first in the sequence of PRs to add manifest file generation (SymlinkInputFormat) to OSS Delta for Presto/Athena read support (issue #76). Specifically, this PR adds the core functionality for manifest generation and rigorous tests to verify the contents of the manifest. Future PRs will add the public APIs for on-demand generation.

• Added post-commit hooks to run tasks after a successful commit.

• Added GenerateSymlinkManifest implementation of post-commit hook to generate the manifests.

  • Each manifest contains the name of data files to read for querying the whole table or partition
  • Non-partitioned table produces a single manifest file containing all the data files.
  • Partitioned table produces partitioned manifest files; same partition structured like the table, each partition directory containing one manifest file containing data files of that partition. This allows Presto/Athena partition-pruned queries to read only manifest files of the necessary partitions.
  • Each attempt to generate manifest will atomically (as much as possible) overwrite the manifest files in the directories (if they exist) and also delete manifest files of partitions that have been deleted from the table.

Co-authored-by: Tathagata Das tathagata.das1565@gmail.com
Co-authored-by: Rahul Mahadev rahul.mahadev@databricks.com

[tdas]

This PR only adds the core functionality, not the public API. If you really want to try it out, then you can generate the manifest by calling the private APIs. ``` val deltaLog = DeltaLog.forTable(spark, tablePath) GenerateSymlinkManifest.generateFullManifest(spark, deltaLog) ``` This will create the manifest file(s) in the directory "tablePath/_symlink_format_manifest/"

…

On Mon, Nov 18, 2019 at 8:58 AM rubinder ***@***.***> wrote: What API does one need to invoke to create the manifest file along with the output file? Do I need to convert it to a Delta Table along the way? I built a jar off of this branch and included it as a dependency to my spark job. When saving the dataframe the code I am using looks like this " outputDataframe .write.format("delta") .mode(SaveMode.Append) .partitionBy(jobArgument.getPartitions: _*) .save(jobArgument.getDestinationPath) " The output is a .snappy.parquet file but I do not see a Manifest File — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#250?email_source=notifications&email_token=AAFB5LBCBIRKFUJFKAUSZ3DQULCUNA5CNFSM4JMMZXMKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEELETUY#issuecomment-555108819>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAFB5LBWV5RVHOCRUEQOOA3QULCUNANCNFSM4JMMZXMA> .