Skip to content

[Fix-9124,9127][Task]Fix the exception of task getting external resource data during execu…#9134

Merged
zhuangchong merged 1 commit into
apache:devfrom
zhuangchong:dev-fix-task-resource
Mar 23, 2022
Merged

[Fix-9124,9127][Task]Fix the exception of task getting external resource data during execu…#9134
zhuangchong merged 1 commit into
apache:devfrom
zhuangchong:dev-fix-task-resource

Conversation

@zhuangchong
Copy link
Copy Markdown
Contributor

@zhuangchong zhuangchong commented Mar 23, 2022

…tion

Purpose of the pull request

this pr closes: #9124 #9127

When the task is executed (such as sql task), the data source or UDF information needs to be used, but the data source ID is stored in the task parameter. At this time, the data source link data needs to be completed according to the ID.

As we all know, the worker node is not connected to the database and cannot obtain other information of the data source based on the data source ID. This part of the operation can only be handled by the master.

Current processing flow:

1.The master calls getResources, and obtains the type and ID of the external resource that needs to be filled according to the task parameters

2.master BaseTaskProcessor.setTaskResourceInfo queries the system library according to the results returned in the previous part, fills in the external resource link data, and stores it in the TaskExecutionContext.resourceParametersHelper property

3.When the task is executed, get the external resource link data from taskExecutionContext.getResourceParametersHelper()


在task执行时候(例如sql任务),需要使用数据源或UDF信息,但是task 参数里面存储的是数据源ID, 这个时候就需要根据ID补齐数据源链接数据。

众所周知,worker 节点是没有连接数据库,不能根据数据源ID去获取数据源其他信息,这部分操作只能由master来负责。

当前的处理流程:

1.master 调用getResources, 根据任务参数获取需要填充外部资源的类型和ID
2.master BaseTaskProcessor.setTaskResourceInfo根据上部分返回的结果查询系统库,填充外部资源链接数据,存放到TaskExecutionContext.resourceParametersHelper属性
3.task 执行时,从taskExecutionContext.getResourceParametersHelper()中获取外部资源链接数据

Brief change log

Verify this pull request

This pull request is code cleanup without any test coverage.

(or)

This pull request is already covered by existing tests, such as (please describe tests).

(or)

This change added tests and can be verified as follows:

Copy link
Copy Markdown
Contributor

@lenboo lenboo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@sonarqubecloud
Copy link
Copy Markdown

Kudos, SonarCloud Quality Gate passed!    Quality Gate passed

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot A 0 Security Hotspots
Code Smell A 0 Code Smells

0.0% 0.0% Coverage
0.0% 0.0% Duplication

@zhuangchong zhuangchong merged commit 25fc1dc into apache:dev Mar 23, 2022
@zhuangchong zhuangchong deleted the dev-fix-task-resource branch March 23, 2022 11:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] [Task Run] PROCEDURE task run fail

3 participants