[ZEPPELIN-1246] In JDBCInterpreter.getScheduler, use getMaxConcurrentConnection instead of hardcoding maxConcurrency to 10

[prasadwagle]

What is this PR for?

In JDBCInterpreter.getScheduler, use getMaxConcurrentConnection instead of hardcoding maxConcurrency to 10

What type of PR is it?

Improvement

Todos

• - Task

What is the Jira issue?

ZEPPELIN-1246

How should this be tested?

Set zeppelin.jdbc.concurrent.max_connection to 2 and verify after starting two long-running queries, a third query is pending.

Screenshots (if appropriate)

Questions:

• Does the licenses files need update?
  No
• Is there breaking changes for older versions?
  No
• Does this needs documentation?
  No

[jongyoul]

Could you please rebase it? it would be fixed yesterday.

[prasadwagle]

I rebased my branch of latest master and pushed it to origin, closed and reopened pull request but the travis build hasn't started. What is the reason for that?

tw-172-25-130-195 incubator-zeppelin-prasadwagle (ZEPPELIN-1246) $ git rebase master
First, rewinding head to replay your work on top of it...
Applying: In JDBCInterpreter.getScheduler, use getMaxConcurrentConnection instead of hardcoding maxConcurrency to 10

tw-172-25-130-195 incubator-zeppelin-prasadwagle (ZEPPELIN-1246) $ git push -f origin
Username for 'https://github.com': prasadwagle
Password for 'https://prasadwagle@github.com': 
Counting objects: 215, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (115/115), done.
Writing objects: 100% (215/215), 188.52 KiB | 0 bytes/s, done.
Total 215 (delta 91), reused 168 (delta 62)
To https://github.com/prasadwagle/incubator-zeppelin.git
 + 4b2990f...62affdf ZEPPELIN-1246 -> ZEPPELIN-1246 (forced update)
tw-172-25-130-195 incubator-zeppelin-prasadwagle (ZEPPELIN-1246) $ git log
commit 62affdfd77a03f597acfc304280d8c40f3a796af
Author: Prasad Wagle <pwagle@twitter.com>
Date:   Thu Jul 28 14:27:15 2016 -0700

    In JDBCInterpreter.getScheduler, use getMaxConcurrentConnection instead of hardcoding maxConcurrency to 10

commit 6f867ceb0c2b93ecca3be8a16c155547182eb0dc
Author: paulbustios <pbustios@gmail.com>
Date:   Mon Aug 1 01:13:27 2016 -0300

[prasadwagle]

Latest build failing with:

T E S T S

Running org.apache.zeppelin.scheduler.ParallelSchedulerTest
20:19:33,549 INFO org.apache.zeppelin.scheduler.SchedulerFactory:131 - Job job1 started by scheduler test
20:19:33,552 INFO org.apache.zeppelin.scheduler.SchedulerFactory:131 - Job job2 started by scheduler test
20:19:34,060 INFO org.apache.zeppelin.scheduler.SchedulerFactory:137 - Job job1 finished by scheduler test
20:19:34,061 INFO org.apache.zeppelin.scheduler.SchedulerFactory:137 - Job job2 finished by scheduler test
20:19:34,061 INFO org.apache.zeppelin.scheduler.SchedulerFactory:131 - Job job3 started by scheduler test
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.881 sec - in org.apache.zeppelin.scheduler.ParallelSchedulerTest
Running org.apache.zeppelin.scheduler.RemoteSchedulerTest
20:19:34,369 INFO org.apache.zeppelin.interpreter.remote.RemoteInterpreterManagedProcess:116 - Run interpreter process [/home/travis/build/apache/zeppelin/zeppelin-interpreter/../bin/interpreter.sh, -d, fake, -p, 42988, -l, fakeRepo]
20:19:34,566 INFO org.apache.zeppelin.scheduler.SchedulerFactory:137 - Job job3 finished by scheduler test
20:19:35,014 INFO org.apache.zeppelin.interpreter.remote.RemoteInterpreter:223 - Create remote interpreter org.apache.zeppelin.interpreter.remote.mock.MockInterpreterA
20:19:35,404 ERROR org.apache.zeppelin.interpreter.remote.RemoteInterpreter:237 - Failed to create interpreter: org.apache.zeppelin.interpreter.remote.mock.MockInterpreterA
20:19:35,405 ERROR org.apache.zeppelin.interpreter.remote.RemoteInterpreter:264 - Failed to initialize interpreter: org.apache.zeppelin.interpreter.remote.mock.MockInterpreterA. Remove it from interpreterGroup
20:19:35,413 INFO org.apache.zeppelin.scheduler.SchedulerFactory:131 - Job jobName1 started by scheduler test
20:19:35,414 INFO org.apache.zeppelin.interpreter.remote.RemoteInterpreter:223 - Create remote interpreter org.apache.zeppelin.interpreter.remote.mock.MockInterpreterA
20:19:35,417 ERROR org.apache.zeppelin.interpreter.remote.RemoteInterpreter:237 - Failed to create interpreter: org.apache.zeppelin.interpreter.remote.mock.MockInterpreterA
20:19:35,418 ERROR org.apache.zeppelin.scheduler.Job:189 - Job failed
org.apache.zeppelin.interpreter.InterpreterException: org.apache.thrift.TApplicationException: Internal error processing createInterpreter
at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.init(RemoteInterpreter.java:238)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getFormType(RemoteInterpreter.java:383)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.interpret(RemoteInterpreter.java:299)
at org.apache.zeppelin.scheduler.RemoteSchedulerTest$2.jobRun(RemoteSchedulerTest.java:210)
at org.apache.zeppelin.scheduler.Job.run(Job.java:176)
at org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:329)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.thrift.TApplicationException: Internal error processing createInterpreter
at org.apache.thrift.TApplicationException.read(TApplicationException.java:111)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:71)
at org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Client.recv_createInterpreter(RemoteInterpreterService.java:196)
at org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Client.createInterpreter(RemoteInterpreterService.java:180)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.init(RemoteInterpreter.java:227)
... 12 more

@jongyoul @Leemoonsoo @bzz - is this a known issue?

[jongyoul]

That error is handling with #1261

[prasadwagle]

I got the latest master with #1261 Two checks in the last build failed. I am going to close and reopen PR to restart build.

https://travis-ci.org/apache/zeppelin/jobs/149535134

Failed tests:
ZeppelinSparkClusterTest.zRunTest:205 expected:<[10]> but was:<[

:28: error: not found: value a
print(a)
^
]>

https://travis-ci.org/apache/zeppelin/jobs/149535133

java.lang.NullPointerException
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterEventPoller.run(RemoteInterpreterEventPoller.java:174)

No output has been received in the last 10 minutes, this potentially indicates a stalled build or something wrong with the build itself.

The build has been terminated

[jongyoul]

@prasadwagle Thanks for fighting the CI. We're also struggling against CI flaky or failed tests

[prasadwagle]

@Leemoonsoo @bzz @jongyoul The build is finally green! Is it ok to merge this change?

[bzz]

Looks great to me!

Thank you @prasadwagle for patience and taking extra care.

Merging to master, if there is no further discussion.