Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

UDF registration does not allow registration for mulitple SparkSessions #1816

Closed
benedeki opened this issue Jun 14, 2021 · 0 comments · Fixed by #1817
Closed

UDF registration does not allow registration for mulitple SparkSessions #1816

benedeki opened this issue Jun 14, 2021 · 0 comments · Fixed by #1817
Assignees
@benedeki
Labels
bug Something isn't working Conformance Conformance Job affected priority: low Nice to have Standardization Standardization Job affected

Comments

@benedeki
Copy link
Collaborator

Describe the bug

The way, how UDF registration is done in Enceladus prevents registering the function in multiple Spark sessions.
It is unlikely to have more, but hard to explain if so.

To Reproduce

Steps to reproduce the behavior:

  1. Create two (or more) Spark sessions
  2. Try to register UD Functions in UDFLibrary for both sessions

Expected behaviour

Possible to register UDF for both sessions
@benedeki benedeki added bug Something isn't working Conformance Conformance Job affected Standardization Standardization Job affected priority: low Nice to have labels Jun 14, 2021
@benedeki benedeki self-assigned this Jun 14, 2021
benedeki added a commit that referenced this issue Jun 14, 2021
@benedeki
#1816: UDF registration does not allow registration for mulitple Spar…
4d22ade 
…kSessions

* Registration is governed by the SparkSession's hash not a simple boolean
benedeki added a commit that referenced this issue Aug 19, 2021
@benedeki
#1816: UDF registration does not allow registration for mulitple Spar…
ac5c0d3 
…kSessions

* Simplified UDFLibrary registration by using `ConcurrentHashMap`
* Refactored `ArrayTransformations`
** Cleaner code
** More Scala like
** Renamed some `vals` for better understanding hopefully
** More is needed...
@benedeki benedeki linked a pull request Sep 7, 2021 that will close this issue
#1816: UDF registration does not allow registration for mulitple Spar… #1817
Merged
benedeki added a commit that referenced this issue Oct 26, 2021
@benedeki
#1816: UDF registration does not allow registration for mulitple Spar… (
3230a6a 
#1817)

* Registration is governed by the SparkSession's hash not a simple boolean
* Refactored `ArrayTransformations`
* Cleaner, more Scala style code
@benedeki benedeki closed this as completed Nov 3, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@benedeki benedeki

Labels
bug Something isn't working Conformance Conformance Job affected priority: low Nice to have Standardization Standardization Job affected
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

#1816: UDF registration does not allow registration for mulitple Spar… AbsaOSS/enceladus
1 participant
@benedeki