Skip to content

fix(ci): restore parallel TileLib ST and watchdog labels - #1105

Merged
mouliangyu merged 2 commits into
hw-native-sys:mainfrom
mouliangyu:codex/fix-tilelib-st-watchdog
Aug 3, 2026
Merged

fix(ci): restore parallel TileLib ST and watchdog labels#1105
mouliangyu merged 2 commits into
hw-native-sys:mainfrom
mouliangyu:codex/fix-tilelib-st-watchdog

Conversation

@mouliangyu

@mouliangyu mouliangyu commented Aug 3, 2026

Copy link
Copy Markdown
Collaborator

背景

PR #1028 已经将 TileLib ST 改为按 case 使用 pytest-xdist 并行调度,但后续 CI workflow 修改重新使用了串行 run_tilelib_st.py 入口。当前 154 个 case 因此共享一个 simulator 会话串行执行;最近一次串行门禁中的 Run TileLib ST CI 用时约 59 分钟。

同一次门禁超过 watchdog 的 90 分钟软预算后,watchdog 虽然正确识别了慢运行,却在给 PR 添加 ci-slow label 时因为 GITHUB_TOKEN 只有 pull-request read 权限而收到 403。该 job 的 continue-on-error 又让 Watchdog workflow 最终显示为 success,没有留下 label 或评论。

修改

  • 恢复 TileLib ST 的 pytest 入口,使用 16 路 pytest-xdist --dist=worksteal 调度独立 case。
  • 保留当前 TILELIB_ST_WORKSPACE、PTODSL Python、PTOAS 和日志路径配置。
  • 将 TileLib artifact 从整个 case simulator 工作目录收敛为聚合日志、每 case 日志、退出码、复现脚本和 msprof stdout,避免并行输出产生百万级临时文件上传。
  • 为慢 CI 告警 job 同时授予 issues: writepull-requests: write,使 GITHUB_TOKEN 可以更新 PR 的 ci-slow label 和 Conversation 评论。
  • 移除 watchdog job 的 continue-on-error,使 watchdog 自身的权限或脚本错误在独立 workflow 中可见;该 workflow 不是 ci-sim required check,不会改变原门禁结果。

验证

  • .github/workflows/ci_sim.yml.github/workflows/watchdog.yml 通过 PyYAML 解析。
  • TileLib ST Python 文件通过 py_compile
  • 使用 CI 同款 PTODSL Python、pytest 9.1.1 和 pytest-xdist 3.8.0:154 个 case 收集成功。
  • pytest --collect-only -n 16 --dist=worksteal test/tilelib-st/test_tilelib_st.py 通过。
  • PR CI 中 154 个 TileLib ST case 全部通过,并行运行时间为 9 分 18 秒。
  • 根据该次 CI 实际输出验证 artifact 收敛:上传文件从 1,101,864 个降至 616 个,约 2.6 MB。
  • git diff --check 通过。

后续 CI 出现的 exit 143 已确认来自宿主机 OOM:PR #1105 的 TileLib simulator 与 PR #1104 的 PTOAS 编译等任务同时运行,runner 所在 tmux scope 使用 OOMPolicy=stop,action 子进程成为 OOM victim 后连带关闭了整个 runner。该宿主机部署问题不通过降低本 PR 的 TileLib 并行度处理。

说明:workflow_run 会从默认分支加载 watchdog,因此新的 PR 写权限需要在本 PR 合入后,由下一次 PR ci-sim 完成事件做端到端验证。

Follow-up to #1028 and #1031.

@mouliangyu mouliangyu changed the title fix(ci): restore parallel TileLib ST and watchdog labels fix(ci): restore parallel TileLib ST and stabilize simulator CI Aug 3, 2026
@mouliangyu mouliangyu changed the title fix(ci): restore parallel TileLib ST and stabilize simulator CI fix(ci): restore parallel TileLib ST and watchdog labels Aug 3, 2026
@mouliangyu
mouliangyu force-pushed the codex/fix-tilelib-st-watchdog branch from 5ebae15 to 31afbbb Compare August 3, 2026 10:52
@mouliangyu
mouliangyu marked this pull request as ready for review August 3, 2026 10:54
@mouliangyu
mouliangyu merged commit dbfbec2 into hw-native-sys:main Aug 3, 2026
9 of 16 checks passed
@mouliangyu
mouliangyu deleted the codex/fix-tilelib-st-watchdog branch August 3, 2026 10:54
@reedhecre

Copy link
Copy Markdown

A5 板测成功

  • 触发方式:merged
  • 源码提交:dbfbec2dcf5e
  • 结果汇总:OK 21 / FAIL 0 / SKIP 0
  • 日志:/root/ptoas-board-monitor-a5/logs/20260803_185605_merged_pr1105.log
  • 结果 TSV:/root/ptoas-board-monitor-a5/logs/20260803_185605_merged_pr1105.tsv

@reedhecre

Copy link
Copy Markdown

A3 板测失败

  • 触发方式:merged
  • 源码提交:dbfbec2dcf5e
  • 结果汇总:OK 218 / FAIL 2 / SKIP 3
  • 日志:/home/zhongxuan/ptoas-board-monitor/runtime/logs/20260803_035525_merged_pr1105.log
  • 失败阶段:board-validation / exit=1

失败用例

  • comm_p2p_binding_variants (run, exit=1)
  • comm_p2p (run, exit=1)

@reedhecre

Copy link
Copy Markdown

A3 板测失败详情:PR #1105

comm_p2p_binding_variants

stage=run info=exit=1

[ERROR] aclrtSynchronizeStream(stream) failed: 507035 (/home/zhongxuan/ptoas-board-monitor/runtime/runs/20260803_035525_merged_pr1105/npu_validation/CommSync/comm_p2p_binding_variants/main.cpp:122)
[ERROR] RecentErrMsg: EZ9999: Inner Error!
EZ9999[PID: 474747] 2026-08-03-04:29:42.876.605 (EZ9999):  The error from device(chipId:1, dieId:1), serial number is 617, there is an exception of aivec error, core id is 46, error code = 0x10, dump info: pc start: 0x124600000000, current: 0x124600000164, vec error info: 0xf7029bb145, mte error info: 0x7503000008, ifu error info: 0x212d052480440, ccu error info: 0x40a414030000004f, cube error info: 0, biu error info: 0, aic error mask: 0x6500020bd00028c, para base: 0x12c100000000.[FUNC:PrintCoreInfo][FILE:device_error_core_proc.cc][LINE:645]
        TraceBack (most recent call last):
       The extend info: errcode:(0x10, 0, 0) errorStr: Illegal instruction, which is usually caused by unaligned UUB addresses. fixp_error0 info: 0x3000008, fixp_error1 info: 0x75, fsmId:0, tslot:3, thread:0, ctxid:0, blk:0, sublk:0, subErrType:4.[FUNC:PrintCoreInfo][FILE:device_error_core_proc.cc][LINE:658]
       Kernel task happen error, retCode=0x31, [vector core exception].[FUNC:PreCheckTaskErr][FILE:davinci_kernel_task.cc][LINE:1729]
       AIV Kernel happen error, retCode=0x31.[FUNC:GetError][FILE:stream.cc][LINE:1475]
       [AIC_INFO] after execute:args print end[FUNC:GetError][FILE:stream.cc][LINE:1475]
       [DFX_INFO]Aicore kernel execute failed, device_id=3, stream_id=46, report_stream_id=46, task_id=0, flip_num=0, fault kernel_name=comm_p2p_binding_variants_kernel, fault kernel info ext=comm_p2p_binding_variants_kernel, program id=0, hash=16541545288638093324.[FUNC:GetError][FILE:stream.cc][LINE:1475]
       rtStreamSynchronize execution failed, reason=vector core exception[FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:65]
       synchronize stream failed, runtime result = 507035[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:148]
[2026-08-03 04:29:44] ERROR: testcase failed (exit 1): comm_p2p_binding_variants
comm_p2p

stage=run info=exit=1

[ERROR] aclrtSynchronizeStream(stream) failed: 507035 (/home/zhongxuan/ptoas-board-monitor/runtime/runs/20260803_035525_merged_pr1105/npu_validation/CommSync/comm_p2p/main.cpp:122)
[ERROR] RecentErrMsg: EZ9999: Inner Error!
EZ9999[PID: 487219] 2026-08-03-04:30:03.343.379 (EZ9999):  The error from device(chipId:1, dieId:1), serial number is 618, there is an exception of aivec error, core id is 1, error code = 0x10, dump info: pc start: 0x124600000000, current: 0x124600000168, vec error info: 0xc9000000bd, mte error info: 0x7503000008, ifu error info: 0x212cf84100200, ccu error info: 0x40a414030000004f, cube error info: 0, biu error info: 0, aic error mask: 0x6500020bd00028c, para base: 0x12c100000000.[FUNC:PrintCoreInfo][FILE:device_error_core_proc.cc][LINE:645]
        TraceBack (most recent call last):
       The extend info: errcode:(0x10, 0, 0) errorStr: Illegal instruction, which is usually caused by unaligned UUB addresses. fixp_error0 info: 0x3000008, fixp_error1 info: 0x75, fsmId:1, tslot:3, thread:0, ctxid:0, blk:0, sublk:0, subErrType:4.[FUNC:PrintCoreInfo][FILE:device_error_core_proc.cc][LINE:658]
       Kernel task happen error, retCode=0x31, [vector core exception].[FUNC:PreCheckTaskErr][FILE:davinci_kernel_task.cc][LINE:1729]
       AIV Kernel happen error, retCode=0x31.[FUNC:GetError][FILE:stream.cc][LINE:1475]
       [AIC_INFO] after execute:args print end[FUNC:GetError][FILE:stream.cc][LINE:1475]
       [DFX_INFO]Aicore kernel execute failed, device_id=3, stream_id=46, report_stream_id=46, task_id=0, flip_num=0, fault kernel_name=comm_p2p_kernel, fault kernel info ext=comm_p2p_kernel, program id=0, hash=6260346515940177617.[FUNC:GetError][FILE:stream.cc][LINE:1475]
       rtStreamSynchronize execution failed, reason=vector core exception[FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:65]
       synchronize stream failed, runtime result = 507035[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:148]
[2026-08-03 04:30:05] ERROR: testcase failed (exit 1): comm_p2p

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants