[SPARK-42169] Implement code generation for to_csv function (StructsToCsv)

[NarekDW]

What changes were proposed in this pull request?

This PR enhances StructsToCsv class with doGenCode function instead of extending it from CodegenFallback trait (performance improvement).

Why are the changes needed?

It will improve performance.

Does this PR introduce any user-facing change?

No

How was this patch tested?

by existing tests

[AmplabJenkins]

Can one of the admins verify this patch?

[HyukjinKwon]

Hm, is the issue from non-codegen part from Rand?

[HyukjinKwon]

Would be easier to review if you describe how it happens and how this PR fixes.

[NarekDW]

Hi @HyukjinKwon I've added 2 sections in the header description - Debugging the root cause and How this PR changes such a behaviour with my point of view, how it happens and how this PR could improve the behaviour.
Could you review, please.

[HyukjinKwon]

cc @cloud-fan FYI

[gboo-infa]

While this change may be good to have for its own reasons, it doesn't really address the problem in the JIRA. The problem is that CodegenFallback is incompatible with nondeterministic child nodes; the problem is not that to_csv happens to be CodegenFallback. Other nodes are also CodegenFallback, and future nodes will be as well, and this change doesn't address that...

[NarekDW]

While this change may be good to have for its own reasons, it doesn't really address the problem in the JIRA. The problem is that CodegenFallback is incompatible with nondeterministic child nodes; the problem is not that to_csv happens to be CodegenFallback. Other nodes are also CodegenFallback, and future nodes will be as well, and this change doesn't address that...

Yep, good point. This is just a partial solution, for to_csv function in particular.
But I don't know, should we rename this PR or create another task/bug in JIRA...

[NarekDW]

@HyukjinKwon any suggestion on this?
Should I keep this PR open?

[cloud-fan]

I'm fixing the root cause at #39248

[cloud-fan]

Can we refine the PR description? It still good to have codegen for to_csv, but it's only for performance, as the correctness issue has been fixed by #39248

[NarekDW]

Can we refine the PR description? It still good to have codegen for to_csv, but it's only for performance, as the correctness issue has been fixed by #39248

@cloud-fan Sure, I've renamed this PR (also changed the description). Unfortunately, I don't have an account in the JIRA, and I can't create a separate JIRA task for this.
Also, I left a test case in this PR:

test("SPARK-41049: make to_csv function deterministic") {
...

I think it might be useful, let me know if it is redundant.

[cloud-fan]

Unfortunately, I don't have an account in the JIRA, and I can't create a separate JIRA task for this.

Can you sign up and create an account? The JIRA system allows anyone to register.

[NarekDW]

Unfortunately, I don't have an account in the JIRA, and I can't create a separate JIRA task for this.

Can you sign up and create an account? The JIRA system allows anyone to register.

Unfortunately, it is disabled

I mailed to private@spark.apache.org to get an account created, about 2 weeks ago, but didn't get any response...
If you can suggest anybody, whom I can address with such a question, it will be great.

[NarekDW]

@cloud-fan finally I've got my JIRA account and created a separate ticket for this.
I've closed this PR as I've renamed my branch in accordance with new JIRA ticket.
Please check the new PR here - #39719

[weicm]

@cloud-fan finally I've got my JIRA account and created a separate ticket for this. I've closed this PR as I've renamed my branch in accordance with new JIRA ticket. Please check the new PR here - #39719

Are you getting a JIRA account by sending an email to private@spark.apache.org?