Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SPARK-34990][SQL][TESTS] Add ParquetEncryptionSuite #32146

Closed
wants to merge 14 commits into from
Closed

[SPARK-34990][SQL][TESTS] Add ParquetEncryptionSuite #32146

wants to merge 14 commits into from

Commits on Apr 22, 2021

  1. [SPARK-34990] Add a test for Parquet Modular Encryption in Spark. 

    A simple test that writes and reads an encrypted parquet
and verifies that it's encrypted by checking its magic string.
    @andersonm-ibm
    andersonm-ibm committed Apr 22, 2021
    Configuration menu
    Copy the full SHA
    3f05a6e View commit details
    Browse the repository at this point in the history

  2. scalastyle

    @andersonm-ibm
    andersonm-ibm committed Apr 22, 2021
    Configuration menu
    Copy the full SHA
    a8b6191 View commit details
    Browse the repository at this point in the history

  3. Address Gidon's comments - reference to VaultClient and remove AAD.

    @andersonm-ibm
    andersonm-ibm committed Apr 22, 2021
    Configuration menu
    Copy the full SHA
    876ec70 View commit details
    Browse the repository at this point in the history

  4. Remove extra spaces in pom.xml

    @andersonm-ibm
    andersonm-ibm committed Apr 22, 2021
    Configuration menu
    Copy the full SHA
    cafe027 View commit details
    Browse the repository at this point in the history

  5. Address review comments - mainly rename test, remove prints and compa… 

    …re result to expected.
    @andersonm-ibm
    andersonm-ibm committed Apr 22, 2021
    Configuration menu
    Copy the full SHA
    8ecee1a View commit details
    Browse the repository at this point in the history

  6. Fix style

    @andersonm-ibm
    andersonm-ibm committed Apr 22, 2021
    Configuration menu
    Copy the full SHA
    0d9c98a View commit details
    Browse the repository at this point in the history

  7. Move parquet encryption test to hive module to remove the old jackson… 

    … dependency from sql module
    @andersonm-ibm
    andersonm-ibm committed Apr 22, 2021
    Configuration menu
    Copy the full SHA
    a83d0ec View commit details
    Browse the repository at this point in the history

  8. Fixed import order

    @andersonm-ibm
    andersonm-ibm committed Apr 22, 2021
    Configuration menu
    Copy the full SHA
    48e8a7a View commit details
    Browse the repository at this point in the history

  9. Use withSQLConf to set hadoop configuration for the test

    @andersonm-ibm
    andersonm-ibm committed Apr 22, 2021
    Configuration menu
    Copy the full SHA
    20df92b View commit details
    Browse the repository at this point in the history

  10. Use InMemoryKms from parquet-hadoop test jar, instead of re-implement… 

    …ing it.
    @andersonm-ibm
    andersonm-ibm committed Apr 22, 2021
    Configuration menu
    Copy the full SHA
    5e3f76c View commit details
    Browse the repository at this point in the history

  11. Read magic string using RandomAccessFile instead of an external process.

    @andersonm-ibm
    andersonm-ibm committed Apr 22, 2021
    Configuration menu
    Copy the full SHA
    4fe0dcb View commit details
    Browse the repository at this point in the history

  12. Change test to use TestHiveSingleton like other tests in Hive module … 

    …instead of SharedSparkSession
    @andersonm-ibm
    andersonm-ibm committed Apr 22, 2021
    Configuration menu
    Copy the full SHA
    3bb9f60 View commit details
    Browse the repository at this point in the history

  13. Use Charset

    @andersonm-ibm
    andersonm-ibm committed Apr 22, 2021
    Configuration menu
    Copy the full SHA
    aa507a4 View commit details
    Browse the repository at this point in the history

Commits on Apr 24, 2021

  1. Add a comment, verify the first magic string instead of the last one,… 

    … verify all parquet parts in parquet folder.
    @andersonm-ibm
    andersonm-ibm committed Apr 24, 2021
    Configuration menu
    Copy the full SHA
    599c71d View commit details
    Browse the repository at this point in the history