Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feature: job拉取GSE文件任务执行日志，支持根据ip查询，减少结果数据 #224

Closed
wangyu096 opened this issue Sep 13, 2021 · 0 comments
Closed

feature: job拉取GSE文件任务执行日志，支持根据ip查询，减少结果数据 #224

wangyu096 opened this issue Sep 13, 2021 · 0 comments
Assignees
@wangyu096
Labels
kind/enhancement 功能改进特性 priority/critical-urgent Highest priority. Must be actively worked on as someone's top priority right now. stage/dev Dev environment in tencent has been deploy stage/prod Production environment in tencent has been deploy stage/test Test environment in tencent has been deploy stage/uat UAT environment in tencent has been deploy
Milestone
V3.4.1

Comments

@wangyu096
Copy link
Collaborator

需求描述：
job当前使用gseTaskId全量查询整个文件任务的执行结果，即使在大部分文件分发子任务都已经完成的情况下。这样会导致两个问题：

  1. 查询全量数据，会占用带宽和影响API调用性能。
  2. 如果一次数据量过大，会导致jvm无法GC

目标：

  1. 根据ip分批查询，避免数据量过大
  2. 支持增量查询，仅查询未完成的任务结果
@wangyu096 wangyu096 added the kind/enhancement 功能改进特性 label Sep 13, 2021
@wangyu096 wangyu096 added this to the V3.4.1 milestone Sep 13, 2021
@wangyu096 wangyu096 self-assigned this Sep 13, 2021
@nekzhang nekzhang added the priority/critical-urgent Highest priority. Must be actively worked on as someone's top priority right now. label Sep 13, 2021
@wangyu096 wangyu096 changed the title feature: job拉取GSE文件任务执行日志，支持分批查询、增量查询 feature: job拉取GSE文件任务执行日志，支持增量查询 Sep 27, 2021
@wangyu096 wangyu096 changed the title feature: job拉取GSE文件任务执行日志，支持增量查询 feature: job拉取GSE文件任务执行日志，支持根据ip查询，减少结果数据 Sep 27, 2021
@wangyu096 wangyu096 added the stage/dev Dev environment in tencent has been deploy label Sep 30, 2021
wangyu096 added a commit to wangyu096/bk-job that referenced this issue Oct 2, 2021
@wangyu096
feature: job拉取GSE文件任务执行日志，支持根据ip查询，减少结果数据 TencentBlueKing#224
fcab3ae 
根据ip查询文件执行结果
wangyu096 added a commit to wangyu096/bk-job that referenced this issue Oct 2, 2021
@wangyu096
feature: job拉取GSE文件任务执行日志，支持根据ip查询，减少结果数据 TencentBlueKing#224
7b75a8e 
根据ip查询文件执行结果
wangyu096 added a commit to wangyu096/bk-job that referenced this issue Oct 2, 2021
@wangyu096
feature: job拉取GSE文件任务执行日志，支持根据ip查询，减少结果数据 TencentBlueKing#224
b279b2e 
根据ip查询文件执行结果
wangyu096 added a commit to wangyu096/bk-job that referenced this issue Oct 2, 2021
@wangyu096
feature: job拉取GSE文件任务执行日志，支持根据ip查询，减少结果数据 TencentBlueKing#224
1044ed5 
过滤已经更新的任务，不再更新DB,提升性能
wangyu096 added a commit to wangyu096/bk-job that referenced this issue Oct 2, 2021
@wangyu096
feature: job拉取GSE文件任务执行日志，支持根据ip查询，减少结果数据 TencentBlueKing#224
32d0529 
按业务灰度
wangyu096 added a commit to wangyu096/bk-job that referenced this issue Oct 2, 2021
@wangyu096
feature: job拉取GSE文件任务执行日志，支持根据ip查询，减少结果数据 TencentBlueKing#224
e6d6f40 
按业务灰度
wangyu096 added a commit to wangyu096/bk-job that referenced this issue Oct 2, 2021
@wangyu096
feature: job拉取GSE文件任务执行日志，支持根据ip查询，减少结果数据 TencentBlueKing#224
13a1802 
按业务灰度
wangyu096 added a commit to wangyu096/bk-job that referenced this issue Oct 2, 2021
@wangyu096
feature: job拉取GSE文件任务执行日志，支持根据ip查询，减少结果数据 TencentBlueKing#224
72c1b02 
按业务灰度
wangyu096 added a commit to wangyu096/bk-job that referenced this issue Oct 2, 2021
@wangyu096
feature: job拉取GSE文件任务执行日志，支持根据ip查询，减少结果数据 TencentBlueKing#224
757508d 
代码warning整改
wangyu096 added a commit to wangyu096/bk-job that referenced this issue Oct 2, 2021
@wangyu096
feature: job拉取GSE文件任务执行日志，支持根据ip查询，减少结果数据 TencentBlueKing#224
0095301 
源上传任务都完成后，拉取结果需要过滤掉源IP
wangyu096 added a commit to wangyu096/bk-job that referenced this issue Oct 25, 2021
@wangyu096
feature: job拉取GSE文件任务执行日志，支持根据ip查询，减少结果数据 TencentBlueKing#224
138dd88 
根据ip查询文件执行结果
jsonwan added a commit that referenced this issue Oct 25, 2021
@jsonwan
Merge pull request #338 from wangyu096/github_perf/pull_gse_file_result
a265edc 
feature: job拉取GSE文件任务执行日志，支持根据ip查询，减少结果数据 #224
@jsonwan jsonwan added the stage/test Test environment in tencent has been deploy label Oct 25, 2021
wangyu096 added a commit to wangyu096/bk-job that referenced this issue Oct 25, 2021
@wangyu096
feature: job拉取GSE文件任务执行日志，支持根据ip查询，减少结果数据 TencentBlueKing#224
f444b78 
更新requirements
jsonwan added a commit that referenced this issue Oct 25, 2021
@jsonwan
Merge pull request #362 from wangyu096/github_perf/pull_gse_file_result
59fb006 
feature: job拉取GSE文件任务执行日志，支持根据ip查询，减少结果数据 #224
@jsonwan jsonwan added the stage/uat UAT environment in tencent has been deploy label Oct 25, 2021
@bkjob-bot bkjob-bot added the stage/prod Production environment in tencent has been deploy label Nov 9, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@wangyu096 wangyu096

Labels
kind/enhancement 功能改进特性 priority/critical-urgent Highest priority. Must be actively worked on as someone's top priority right now. stage/dev Dev environment in tencent has been deploy stage/prod Production environment in tencent has been deploy stage/test Test environment in tencent has been deploy stage/uat UAT environment in tencent has been deploy
Projects
None yet
Milestone
V3.4.1
Development

No branches or pull requests
4 participants
@nekzhang @wangyu096 @jsonwan @bkjob-bot