Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Register udf for every sparksession #1258

Merged
merged 4 commits into from Nov 27, 2019
Merged

Register udf for every sparksession #1258

merged 4 commits into from Nov 27, 2019

Conversation

@birdstorm
Copy link
Member

birdstorm commented Nov 27, 2019

What problem does this PR solve?

Close #1248

What is changed and how it works?

Register udf for every sparksession

Check List

Tests

  • Manual test (add detailed scripts or steps below)

Follow steps in #1248.

Code changes

  • Has interface methods change
  • Has persistent data change

Side effects

  • Possible performance regression
  • Increased code complexity

Related changes

  • Need to cherry-pick to the release branch
  • Need to be included in the release note
@birdstorm birdstorm requested review from marsishandsome and zhexuany Nov 27, 2019
birdstorm added 3 commits Nov 27, 2019
@birdstorm

This comment has been minimized.

Copy link
Member Author

birdstorm commented Nov 27, 2019

/run-all-tests

Copy link
Member

zhexuany left a comment

LGTM

@zhexuany zhexuany merged commit 0ff164a into master Nov 27, 2019
3 checks passed
3 checks passed
idc-jenkins-ci-tispark/build Jenkins job succeeded.
Details
idc-jenkins-ci-tispark/integration-test Jenkins job succeeded.
Details
license/cla Contributor License Agreement is signed.
Details
@birdstorm birdstorm deleted the fix-udf-initialization branch Nov 27, 2019
Copy link
Collaborator

marsishandsome left a comment

better to add some tests:

1. new two SparkSessions
2. set the current database in one SparkSession
3. get the current database in another
1. new two SparkSessions
2. call ti_version() in both SparkSessions
class TiExtensions extends (SparkSessionExtensions => Unit) {
private val tiContextMap = mutable.HashMap.empty[SparkSession, TiContext]

def getOrCreateTiContext(sparkSession: SparkSession): TiContext = synchronized {

This comment has been minimized.

Copy link
@marsishandsome

marsishandsome Nov 28, 2019

Collaborator

the name TiContext confused me, maybe should change to TiSession

birdstorm added a commit that referenced this pull request Nov 29, 2019
birdstorm added a commit that referenced this pull request Nov 29, 2019
birdstorm added a commit that referenced this pull request Dec 2, 2019
birdstorm added a commit that referenced this pull request Dec 3, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.