[SPARK-35472][PYTHON] Fix disallow_untyped_defs mypy checks for pyspark.pandas.generic

[ueshin]

What changes were proposed in this pull request?

Adds more type annotations in the file python/pyspark/pandas/generic.py and fixes the mypy check failures.

Why are the changes needed?

We should enable more disallow_untyped_defs mypy checks.

Does this PR introduce any user-facing change?

Yes.
This PR adds more type annotations in pandas APIs on Spark module, which can impact interaction with development tools for users.

How was this patch tested?

The mypy check with a new configuration and existing tests should pass.

[SparkQA]

Test build #139955 has finished for PR 32957 at commit d4c3537.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44482/

[SparkQA]

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44482/

[SparkQA]

Test build #139956 has finished for PR 32957 at commit 67703d9.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44483/

[SparkQA]

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44483/

[SparkQA]

Test build #140000 has finished for PR 32957 at commit 847df02.

• This patch fails Python style tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44526/

[ueshin]

Jenkins, retest this please.

[SparkQA]

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44526/

[SparkQA]

Test build #140002 has finished for PR 32957 at commit 847df02.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[ueshin]

cc @HyukjinKwon @itholic @xinrong-databricks

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44528/

[SparkQA]

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44528/

[SparkQA]

Test build #140008 has finished for PR 32957 at commit 4d8a751.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Kubernetes integration test unable to build dist.

exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44534/

[HyukjinKwon]

Merged to master.