[Fix-9124,9127][Task]Fix the exception of task getting external resource data during execu…#9134
Merged
Merged
Conversation
caishunfeng
approved these changes
Mar 23, 2022
|
Kudos, SonarCloud Quality Gate passed! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.








…tion
Purpose of the pull request
this pr closes: #9124 #9127
When the task is executed (such as sql task), the data source or UDF information needs to be used, but the data source ID is stored in the task parameter. At this time, the data source link data needs to be completed according to the ID.
As we all know, the worker node is not connected to the database and cannot obtain other information of the data source based on the data source ID. This part of the operation can only be handled by the master.
Current processing flow:
1.The master calls getResources, and obtains the type and ID of the external resource that needs to be filled according to the task parameters
2.master BaseTaskProcessor.setTaskResourceInfo queries the system library according to the results returned in the previous part, fills in the external resource link data, and stores it in the TaskExecutionContext.resourceParametersHelper property
3.When the task is executed, get the external resource link data from taskExecutionContext.getResourceParametersHelper()
在task执行时候(例如sql任务),需要使用数据源或UDF信息,但是task 参数里面存储的是数据源ID, 这个时候就需要根据ID补齐数据源链接数据。
众所周知,worker 节点是没有连接数据库,不能根据数据源ID去获取数据源其他信息,这部分操作只能由master来负责。
当前的处理流程:
1.master 调用getResources, 根据任务参数获取需要填充外部资源的类型和ID
2.master BaseTaskProcessor.setTaskResourceInfo根据上部分返回的结果查询系统库,填充外部资源链接数据,存放到TaskExecutionContext.resourceParametersHelper属性
3.task 执行时,从taskExecutionContext.getResourceParametersHelper()中获取外部资源链接数据
Brief change log
Verify this pull request
This pull request is code cleanup without any test coverage.
(or)
This pull request is already covered by existing tests, such as (please describe tests).
(or)
This change added tests and can be verified as follows: