[SOT] merge PaddleSOT into Paddle #57824

SigureMo · 2023-09-27T16:02:41Z

PR types

New features

PR changes

Others

Description

将 PaddleSOT 并入 Paddle 主仓库，目录结构对应关系如下：

PaddleSOT/sot -> Paddle/python/paddle/jit/sot
PaddleSOT/tests -> Paddle/test/sot （sot 相关 import 需要修改）

SOT 相关修改：

将环境变量 LOG_LEVEL 改为 SOT_LOG_LEVEL，并删除设置 level 为 2 的操作，现在默认不会再打印很多 log，PTS 现在也支持设置环境变量，合入后和 QA 和新 IR 同学同步环境变量变动即可
修复路径分隔符硬编码为 /，而 Windows 上为 \\ 的问题，导致无法正常 skip paddle API，使用 os.sep 即可

Paddle 相关修改：

PR-CI-SOT 流水线接入合入后的 paddle.jit.sot 而不是 clone 的 PaddleSOT
动转静单测开启 SOT 测试，现在会同时跑 SOT 和原来的 AST 版本的，并设置了相关的环境变量（COST_MODEL 等）
动转静单测添加新的 dygraph_to_static_utils_new，允许细粒度 disable 某一个单测，并方便之后开启 PIR 动转静最终态单测，目前 SOT + PIR 会出现问题，@kangguangli 正在修复（[PIR] fix share_buffer translation #58075），将会在之后打开，目前仅仅针对报错单测开启这个，新旧并行，将会在之后同 @gouzil 推全新的机制
修复 to_tensor 静态图分支在 Windows 上默认 dtype 为 int32 导致的问题

一些 TODOs：

接口形态的确认，是否切默认，使用什么 flag 切换 SOT/AST 模式
to_static 里 import sot 目前在函数中使用了动态 import，之后可以考虑重构部分代码以确保在没有 circular import 的前提下将其放到顶层
单测新机制（dygraph_to_static_utils_new）的推全
SOT 目录结构重构，避免目录结构层次过深的问题
仅修改部分目录才触发 PR-CI-SOT [SOT][ci] add check file run #58051
SOT 单测优化，可以考虑是否接入 ctest，使得运行方式一致（但要保留直接 python 跑的方式）

PCard-66972

PaddleSOT is a Bytecode level Implementation of Symbolic OpCode Translator For PaddlePaddle. We originally developed in [PaddleSOT](https://github.com/PaddlePaddle/PaddleSOT), and to ensure consistency in Paddle versions, we are now merging PaddleSOT into Paddle. Thanks to all the contributors of this project! See more details in https://github.com/PaddlePaddle/PaddleSOT/graphs/contributors --- Co-authored-by: xiongkun <xiongkun03@baidu.com> Co-authored-by: feifei-111 <2364819892@qq.com> Co-authored-by: 0x45f <23097963+0x45f@users.noreply.github.com> Co-authored-by: gouzil <66515297+gouzil@users.noreply.github.com> Co-authored-by: 六个骨头 <46243324+zrr1999@users.noreply.github.com> Co-authored-by: Aurelius84 <zhangliujie@baidu.com> Co-authored-by: Wang Xin <xinwang614@gmail.com> Co-authored-by: haozi <64006169+NotHaozi@users.noreply.github.com> Co-authored-by: RedContritio <RedContritio@qq.com> Co-authored-by: Sanbu <96160062+sanbuphy@users.noreply.github.com> Co-authored-by: Difer <c7070655110@gmail.com> Co-authored-by: cyberslack_lee <luhputu0815@gmail.com> Co-authored-by: jjyaoao <jjyaoao@126.com> Co-authored-by: PuQing <me@puqing.work> Co-authored-by: Ran chongzhi <57489288+ranchongzhi@users.noreply.github.com> Co-authored-by: Zhenghai Zhang <65210872+ccsuzzh@users.noreply.github.com>

paddle-bot · 2023-09-27T16:02:45Z

你的PR提交成功，感谢你对开源项目的贡献!
请关注后续CI自动化测试结果，详情请参考Paddle-CI手册。
Your PR has been submitted. Thanks for your contribution!
Please wait for the result of CI firstly. See Paddle CI Manual for details.

paddle-ci-bot · 2023-10-05T03:14:47Z

Sorry to inform you that 49933ae's CIs have passed for more than 7 days. To prevent PR conflicts, you need to re-run all CIs manually.

XieYunshen

LGTM
单测超时时间设置

risemeup1

LGTM for setup.py

zyfncg

LGTM for setup.py.in

XiaoguangHu01

LGTM

PaddleSOT is a Bytecode level Implementation of Symbolic OpCode Translator For PaddlePaddle. We originally developed in [PaddleSOT](https://github.com/PaddlePaddle/PaddleSOT), and to ensure consistency in Paddle versions, we are now merging PaddleSOT into Paddle. Thanks to all the contributors of this project! See more details in https://github.com/PaddlePaddle/PaddleSOT/graphs/contributors --------- Co-authored-by: xiongkun <xiongkun03@baidu.com> Co-authored-by: feifei-111 <2364819892@qq.com> Co-authored-by: 0x45f <23097963+0x45f@users.noreply.github.com> Co-authored-by: gouzil <66515297+gouzil@users.noreply.github.com> Co-authored-by: 六个骨头 <46243324+zrr1999@users.noreply.github.com> Co-authored-by: Aurelius84 <zhangliujie@baidu.com> Co-authored-by: Wang Xin <xinwang614@gmail.com> Co-authored-by: haozi <64006169+NotHaozi@users.noreply.github.com> Co-authored-by: RedContritio <RedContritio@qq.com> Co-authored-by: Sanbu <96160062+sanbuphy@users.noreply.github.com> Co-authored-by: Difer <c7070655110@gmail.com> Co-authored-by: cyberslack_lee <luhputu0815@gmail.com> Co-authored-by: jjyaoao <jjyaoao@126.com> Co-authored-by: PuQing <me@puqing.work> Co-authored-by: Ran chongzhi <57489288+ranchongzhi@users.noreply.github.com> Co-authored-by: Zhenghai Zhang <65210872+ccsuzzh@users.noreply.github.com>

remove private apis from __all__

f801e01

SigureMo changed the title ~~[SOT] merge PaddleSOT into Paddle~~ [WIP][SOT] merge PaddleSOT into Paddle Sep 27, 2023

remove from . import sot

49933ae

SigureMo added 5 commits October 10, 2023 13:03

Merge branch 'develop' into sot/merge-into-paddle

02b6e07

enable sot tests

5262e83

Merge branch 'develop' into sot/merge-into-paddle

d213e07

update with latest sot

107018d

update with latest sot

a7d42ad

SigureMo marked this pull request as ready for review October 10, 2023 10:05

SigureMo added 18 commits October 10, 2023 20:07

update envs

8a12a36

add @dy2static_unittest

93b08d9

add STRICT_MODE

dc152cb

Merge branch 'develop' into sot/merge-into-paddle

25d14db

update PR-CI-SOT

2d8166d

add sot to setup packages

d110b62

add all sot packages

923b68f

fix test_exception lineno

10057b1

fix some test_and_compare_with_new_ir shouldnt compare

a0f4650

fix patch path

936b91a

rename LOG_LEVEL to SOT_LOG_LEVEL

8452698

add new test utils

ca4efcb

fix more unittests

6a5c82f

increase some test case time

b3c1044

fix coverage

50b0554

set SOT_LOG_LEVEL back to 0

68055bc

fix test_tensor_memcpy_on_gpu

2b01145

fix missing skip on Windows

0cede4d

SigureMo added 2 commits October 13, 2023 03:46

using logger

c0c2e8b

fix to_tensor dtype on Windows

1c519dc

Aurelius84 approved these changes Oct 13, 2023

View reviewed changes

SigureMo mentioned this pull request Oct 13, 2023

add announcement for migration PaddlePaddle/PaddleSOT#420

Merged

SigureMo changed the title ~~[WIP][SOT] merge PaddleSOT into Paddle~~ [SOT] merge PaddleSOT into Paddle Oct 13, 2023

Aurelius84 requested review from tianshuo78520a, risemeup1, zyfncg and XiaoguangHu01 October 13, 2023 11:20

XieYunshen approved these changes Oct 13, 2023

View reviewed changes

Aurelius84 requested review from phlrain and raindrops2sea October 13, 2023 11:21

risemeup1 approved these changes Oct 13, 2023

View reviewed changes

zyfncg approved these changes Oct 13, 2023

View reviewed changes

kangguangli approved these changes Oct 13, 2023

View reviewed changes

zhangbo9674 approved these changes Oct 13, 2023

View reviewed changes

zrr1999 approved these changes Oct 13, 2023

View reviewed changes

XiaoguangHu01 approved these changes Oct 13, 2023

View reviewed changes

SigureMo merged commit a49ee53 into PaddlePaddle:develop Oct 13, 2023
28 checks passed

SigureMo deleted the sot/merge-into-paddle branch October 13, 2023 13:57

This was referenced Oct 13, 2023

🐾 PaddleSOT 快乐喵喵开源任务 🚀 PaddlePaddle/PaddleSOT#133

Closed

🪄 PaddleSOT 喵喵大入侵计划 💥 PaddlePaddle/PaddleSOT#305

Closed

This was referenced Oct 17, 2023

[SOT] enable PIR + SOT dy2st unittest cases #58156

Merged

[Dy2St] Refine dy2st unittest decorators name #58316

Merged

gouzil mentioned this pull request Oct 24, 2023

动转静单测机制推全任务列表 #58356

Closed

gouzil mentioned this pull request Nov 2, 2023

PIR 动转静理想态单测推全验证任务列表（一期） #58633

Closed

ooooo-create mentioned this pull request Nov 24, 2023

动转静单测时间统计和优化专项 #59339

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SOT] merge PaddleSOT into Paddle #57824

[SOT] merge PaddleSOT into Paddle #57824

SigureMo commented Sep 27, 2023 •

edited

Loading

paddle-bot bot commented Sep 27, 2023

paddle-ci-bot bot commented Oct 5, 2023

XieYunshen left a comment

risemeup1 left a comment

zyfncg left a comment

XiaoguangHu01 left a comment

[SOT] merge PaddleSOT into Paddle #57824

[SOT] merge PaddleSOT into Paddle #57824

Conversation

SigureMo commented Sep 27, 2023 • edited Loading

PR types

PR changes

Description

paddle-bot bot commented Sep 27, 2023

paddle-ci-bot bot commented Oct 5, 2023

XieYunshen left a comment

Choose a reason for hiding this comment

risemeup1 left a comment

Choose a reason for hiding this comment

zyfncg left a comment

Choose a reason for hiding this comment

XiaoguangHu01 left a comment

Choose a reason for hiding this comment

SigureMo commented Sep 27, 2023 •

edited

Loading