Skip to content

[python] Fix database error handler about no datasource name#7631

Merged
CalvinKirs merged 3 commits intoapache:devfrom
zhongjiajie:py-bug-core-db-not-exists
Jan 7, 2022
Merged

[python] Fix database error handler about no datasource name#7631
CalvinKirs merged 3 commits intoapache:devfrom
zhongjiajie:py-bug-core-db-not-exists

Conversation

@zhongjiajie
Copy link
Copy Markdown
Member

The original code show too many unrelated log when database name
do not exists. BTW, it would continue the code when error miss.
This patch is try to fix it.

close: #7616

@zhongjiajie zhongjiajie added bug Something isn't working Pyscheduler labels Dec 26, 2021
@zhongjiajie zhongjiajie self-assigned this Dec 26, 2021
@codecov-commenter
Copy link
Copy Markdown

codecov-commenter commented Dec 26, 2021

Codecov Report

Merging #7631 (567039d) into dev (46c39cd) will decrease coverage by 0.02%.
The diff coverage is n/a.

Impacted file tree graph

@@             Coverage Diff              @@
##                dev    #7631      +/-   ##
============================================
- Coverage     41.03%   41.00%   -0.03%     
+ Complexity     3696     3695       -1     
============================================
  Files           637      637              
  Lines         27025    27044      +19     
  Branches       3063     3067       +4     
============================================
  Hits          11090    11090              
- Misses        14867    14885      +18     
- Partials       1068     1069       +1     
Impacted Files Coverage Δ
...inscheduler/api/controller/ExecutorController.java 10.00% <0.00%> (-15.00%) ⬇️
...dolphinscheduler/remote/future/ResponseFuture.java 81.96% <0.00%> (-1.64%) ⬇️
.../org/apache/dolphinscheduler/api/enums/Status.java 100.00% <0.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 46c39cd...567039d. Read the comment docs.

@zhongjiajie zhongjiajie marked this pull request as ready for review January 5, 2022 10:30
@zhongjiajie zhongjiajie force-pushed the py-bug-core-db-not-exists branch from 0821abc to f84047d Compare January 5, 2022 10:30
@zhongjiajie
Copy link
Copy Markdown
Member Author

@devosend PTAL, if you have time, thanks.

The original code show too many unrelated log when database name
do not exists. BTW, it would continue the code when error miss.
This patch is try to fix it.

close: apache#7616
@zhongjiajie zhongjiajie force-pushed the py-bug-core-db-not-exists branch from f84047d to 63de7af Compare January 7, 2022 06:53
@zhongjiajie
Copy link
Copy Markdown
Member Author

I here allow it raise error base on py4j jave gateway server track stack, and I directly using chaining error message for more easy location the error. You could see in keyword During handling of the above exception, another exception occurred

Here it is an example

Traceback (most recent call last):
  File "/Users/zhongjiajie/Documents/program/dolphinscheduler/dolphinscheduler-python/pydolphinscheduler/src/pydolphinscheduler/core/database.py", line 59, in get_database_info
    self._database = gateway.entry_point.getDatasourceInfo(name)
  File "/Users/zhongjiajie/.virtualenvs/pydolphinschduler/lib/python3.8/site-packages/py4j/java_gateway.py", line 1309, in __call__
    return_value = get_return_value(
  File "/Users/zhongjiajie/.virtualenvs/pydolphinschduler/lib/python3.8/site-packages/py4j/protocol.py", line 326, in get_return_value
    raise Py4JJavaError(
py4j.protocol.Py4JJavaError: An error occurred while calling t.getDatasourceInfo.
: java.lang.IllegalArgumentException: Can not find any datasource by name first_mysql
	at org.apache.dolphinscheduler.server.PythonGatewayServer.getDatasourceInfo(PythonGatewayServer.java:398)
	at sun.reflect.GeneratedMethodAccessor187.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
	at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
	at py4j.Gateway.invoke(Gateway.java:282)
	at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
	at py4j.commands.CallCommand.execute(CallCommand.java:79)
	at py4j.GatewayConnection.run(GatewayConnection.java:238)
	at java.lang.Thread.run(Thread.java:748)


During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/zhongjiajie/Documents/program/dolphinscheduler/dolphinscheduler-python/pydolphinscheduler/examples/task_datax_example.py", line 94, in <module>
    pd.run()
  File "/Users/zhongjiajie/Documents/program/dolphinscheduler/dolphinscheduler-python/pydolphinscheduler/src/pydolphinscheduler/core/process_definition.py", line 325, in run
    self.submit()
  File "/Users/zhongjiajie/Documents/program/dolphinscheduler/dolphinscheduler-python/pydolphinscheduler/src/pydolphinscheduler/core/process_definition.py", line 377, in submit
    json.dumps(self.task_definition_json),
  File "/Users/zhongjiajie/Documents/program/dolphinscheduler/dolphinscheduler-python/pydolphinscheduler/src/pydolphinscheduler/core/process_definition.py", line 215, in task_definition_json
    return [task.get_define() for task in self.tasks.values()]
  File "/Users/zhongjiajie/Documents/program/dolphinscheduler/dolphinscheduler-python/pydolphinscheduler/src/pydolphinscheduler/core/process_definition.py", line 215, in <listcomp>
    return [task.get_define() for task in self.tasks.values()]
  File "/Users/zhongjiajie/Documents/program/dolphinscheduler/dolphinscheduler-python/pydolphinscheduler/src/pydolphinscheduler/core/base.py", line 69, in get_define
    content = self.get_define_custom(camel_attr, self._DEFINE_ATTR)
  File "/Users/zhongjiajie/Documents/program/dolphinscheduler/dolphinscheduler-python/pydolphinscheduler/src/pydolphinscheduler/core/base.py", line 56, in get_define_custom
    val = getattr(self, attr, None)
  File "/Users/zhongjiajie/Documents/program/dolphinscheduler/dolphinscheduler-python/pydolphinscheduler/src/pydolphinscheduler/tasks/datax.py", line 116, in task_params
    datasource = Database(self.datasource_name, "dsType", "dataSource")
  File "/Users/zhongjiajie/Documents/program/dolphinscheduler/dolphinscheduler-python/pydolphinscheduler/src/pydolphinscheduler/core/database.py", line 39, in __init__
    self[type_key] = self.database_type
  File "/Users/zhongjiajie/Documents/program/dolphinscheduler/dolphinscheduler-python/pydolphinscheduler/src/pydolphinscheduler/core/database.py", line 45, in database_type
    return self.get_database_info(self.database_name).get("type")
  File "/Users/zhongjiajie/Documents/program/dolphinscheduler/dolphinscheduler-python/pydolphinscheduler/src/pydolphinscheduler/core/database.py", line 62, in get_database_info
    raise PyDSParamException(str(ex.java_exception))
pydolphinscheduler.exceptions.PyDSParamException: java.lang.IllegalArgumentException: Can not find any datasource by name first_mysql

Process finished with exit code 1

@zhongjiajie
Copy link
Copy Markdown
Member Author

PTAL @devosend

@sonarqubecloud
Copy link
Copy Markdown

sonarqubecloud bot commented Jan 7, 2022

Kudos, SonarCloud Quality Gate passed!    Quality Gate passed

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot A 0 Security Hotspots
Code Smell A 0 Code Smells

0.0% 0.0% Coverage
0.0% 0.0% Duplication

@devosend
Copy link
Copy Markdown
Contributor

devosend commented Jan 7, 2022

PTAL @devosend

LGTM @zhongjiajie

Copy link
Copy Markdown
Member

@CalvinKirs CalvinKirs left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@CalvinKirs CalvinKirs merged commit 6532215 into apache:dev Jan 7, 2022
@zhongjiajie zhongjiajie deleted the py-bug-core-db-not-exists branch January 7, 2022 13:29
@zhongjiajie
Copy link
Copy Markdown
Member Author

Thanks

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working Pyscheduler

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] [python] Class database raise error NPE when database name not exists

4 participants