[SPARK-34479][SQL] Add zstandard codec to Avro compression codec list

[wangyum]

What changes were proposed in this pull request?

Avro add zstandard codec since AVRO-2195. This pr add zstandard codec to Avro compression codec list.

Why are the changes needed?

To make Avro support zstandard codec.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Unit test.

[dongjoon-hyun]

Thank you for sharing PR, @wangyum . I believe we can collaborate to fix the issue and merge this PR soon via luben/zstd-jni#161 .

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40116/

[SparkQA]

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40116/

[dongjoon-hyun]

I made a PR. It will unblock this issue.

• [SPARK-34559][BUILD] Upgrade to ZSTD JNI 1.4.8-6 #31674

I verified the new ZSTD JNI library on top of this PR. AvroV1Suite and AvroV2Suite passed.

[dongjoon-hyun]

Could you rebase this PR to the master?

[SparkQA]

Test build #135535 has finished for PR 31673 at commit 136c5b0.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #135541 has finished for PR 31673 at commit 309de76.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[dongjoon-hyun]

+1, LGTM. Thank you!
Merged to master.