Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FLINK-15635][table] EnvironmentSettings supports to accept the user ClassLoader #19845

Merged
merged 9 commits into from Jun 8, 2022

Conversation

lsyldliu
Copy link
Contributor

@lsyldliu lsyldliu commented May 30, 2022

This is a continued work of contribution #19410.

This PR solves the issue of allowing the user to provide its own classloader for UDFs and at other places where it's more appropriate than the current Thread context classloader.

The PR also contains a couple of hotfixes (mergeable separately):

Deprecate CatalogFunction#isGeneric
Remove CodeGeneratorContext#apply which makes the code unnecessarily more complicated
Fix singleton config in EnvironmentSettings, which among various issues, it's also causing in this case to select the wrong user class loader

@flinkbot
Copy link
Collaborator

flinkbot commented May 30, 2022

CI report:

Bot commands The @flinkbot bot supports the following commands:
  • @flinkbot run azure re-run the last Azure build

@lsyldliu lsyldliu changed the title [FLINK-15635][table] Now EnvironmentSettings accepts the user ClassLoader [FLINK-15635][table] EnvironmentSettings supports to accept the user ClassLoader May 30, 2022
@wuchong
Copy link
Member

wuchong commented May 31, 2022

Hi @lsyldliu , it seems that the spotless is failed. Could you fix it?

@lsyldliu
Copy link
Contributor Author

lsyldliu commented Jun 1, 2022

Hi @lsyldliu , it seems that the spotless is failed. Could you fix it?
Thanks for reminder, I have fixed it.

Copy link
Member

@wuchong wuchong left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @slinkydeveloper , @matriv , @lsyldliu , the pull request looks good to me in general. I only left 2 comments.

/**
* Note that this cast context is going to use the thread context classloader. This is fine when
* the context will be used to generate casting for primitive types, but it might create problems
* when dealing with user provided types.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This sounds problematic? Why not use userClassloader here?

val objCopy: AnyRef =
InstantiationUtil.deserializeObject(byteArray, Thread.currentThread().getContextClassLoader)
references += objCopy
references += obj
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @slinkydeveloper , I think the CodeGeneratorContext doesn't know whether the obj would be mutated or not, so it would be safer to clone the obj here. Do you have any concerns to copy the object?

@wuchong
Copy link
Member

wuchong commented Jun 2, 2022

Besides, there are many cases are failed, could you take a look?

Signed-off-by: slinkydeveloper <francescoguard@gmail.com>
…Context

Signed-off-by: slinkydeveloper <francescoguard@gmail.com>
Signed-off-by: slinkydeveloper <francescoguard@gmail.com>
Signed-off-by: slinkydeveloper <francescoguard@gmail.com>
Copy link
Member

@wuchong wuchong left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

@wuchong
Copy link
Member

wuchong commented Jun 7, 2022

There are too many commits. I will squash and tidy the commits.

@wuchong
Copy link
Member

wuchong commented Jun 8, 2022

@lsyldliu could you help to have a look at the test? It seems {{SplitAggregateITCase.testMinMaxWithRetraction}} is failed.

@lsyldliu
Copy link
Contributor Author

lsyldliu commented Jun 8, 2022

@flinkbot run azure

@wuchong
Copy link
Member

wuchong commented Jun 8, 2022

Merging...

@wuchong wuchong merged commit 7aa58b3 into apache:master Jun 8, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
5 participants