[Inductor][fx pass] Fuse pointwise operators in the post grad #114778

mengluy0125 · 2023-11-29T19:23:54Z

Summary: We construct a unified API that can be easily add pointwise ops to be batched in the post grad

Test Plan:

unit test

buck2 test 'fbcode//mode/dev-nosan' fbcode//caffe2/test/inductor:group_batch_fusion

Buck UI: https://www.internalfb.com/buck2/19b3f641-782f-4f94-a953-3ff9ce2cfa7b
Test UI: https://www.internalfb.com/intern/testinfra/testrun/1125900251953016
Network: Up: 67KiB Down: 32KiB (reSessionID-c2a80f26-8227-4f78-89fc-bcbda0ae8353)
Jobs completed: 18. Time elapsed: 1:19.8s.
Cache hits: 0%. Commands: 2 (cached: 0, remote: 0, local: 2)
Tests finished: Pass 6. Fail 0. Fatal 0. Skip 0. Build failure 0

local reproduce

cmf

P881792289

igctr

dsnn

icvr

Reviewed By: xuzhao9

Differential Revision: D51332067

cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @yf225 @chenyang78 @kadeng @muchulee8 @aakhundov @ColinPeppler

pytorch-bot · 2023-11-29T19:23:58Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/114778

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

✅ You can merge normally! (2 Unrelated Failures)

As of commit 1cda761 with merge base a6736ac ():

BROKEN TRUNK - The following job failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

inductor-periodic / cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (dynamic_aot_eager_torchbench, 1, 2, linux.g5.4xlarge.nvidia.gpu) (gh)

UNSTABLE - The following job failed but was likely due to flakiness present on trunk and has been marked as unstable:

trunk / linux-focal-rocm5.7-py3.8 / test (default, 1, 1, linux.rocm.gpu, unstable) (gh)

This comment was automatically generated by Dr. CI and updates every 15 minutes.

facebook-github-bot · 2023-11-29T19:24:06Z

This pull request was exported from Phabricator. Differential Revision: D51332067

github-actions · 2023-11-29T19:25:25Z

This PR needs a `release notes:` label

If your changes are user facing and intended to be a part of release notes, please use a label starting with release notes:.

If not, please add the topic: not user facing label.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "topic: not user facing"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

…h#114778) Summary: We construct a unified API that can be easily add pointwise ops to be batched in the post grad Test Plan: # unit test ``` buck2 test 'fbcode//mode/dev-nosan' fbcode//caffe2/test/inductor:group_batch_fusion ``` Buck UI: https://www.internalfb.com/buck2/19b3f641-782f-4f94-a953-3ff9ce2cfa7b Test UI: https://www.internalfb.com/intern/testinfra/testrun/1125900251953016 Network: Up: 67KiB Down: 32KiB (reSessionID-c2a80f26-8227-4f78-89fc-bcbda0ae8353) Jobs completed: 18. Time elapsed: 1:19.8s. Cache hits: 0%. Commands: 2 (cached: 0, remote: 0, local: 2) Tests finished: Pass 6. Fail 0. Fatal 0. Skip 0. Build failure 0 # local reproduce ### cmf P881792289 ### igctr ### dsnn ### icvr Reviewed By: xuzhao9 Differential Revision: D51332067

…h#114778) Summary: We construct a unified API that can be easily add pointwise ops to be batched in the post grad Test Plan: # unit test ``` buck2 test 'fbcode//mode/dev-nosan' fbcode//caffe2/test/inductor:group_batch_fusion ``` Buck UI: https://www.internalfb.com/buck2/19b3f641-782f-4f94-a953-3ff9ce2cfa7b Test UI: https://www.internalfb.com/intern/testinfra/testrun/1125900251953016 Network: Up: 67KiB Down: 32KiB (reSessionID-c2a80f26-8227-4f78-89fc-bcbda0ae8353) Jobs completed: 18. Time elapsed: 1:19.8s. Cache hits: 0%. Commands: 2 (cached: 0, remote: 0, local: 2) Tests finished: Pass 6. Fail 0. Fatal 0. Skip 0. Build failure 0 # local reproduce ### cmf P887605070 ### igctr P892987433 ### mai P893109069 ### icvr P893075846 Reviewed By: xuzhao9 Differential Revision: D51332067

facebook-github-bot · 2023-11-29T22:55:21Z

This pull request was exported from Phabricator. Differential Revision: D51332067

…h#114778) Summary: We construct a unified API that can be easily add pointwise ops to be batched in the post grad Test Plan: # unit test ``` buck2 test 'fbcode//mode/dev-nosan' fbcode//caffe2/test/inductor:group_batch_fusion ``` Buck UI: https://www.internalfb.com/buck2/19b3f641-782f-4f94-a953-3ff9ce2cfa7b Test UI: https://www.internalfb.com/intern/testinfra/testrun/1125900251953016 Network: Up: 67KiB Down: 32KiB (reSessionID-c2a80f26-8227-4f78-89fc-bcbda0ae8353) Jobs completed: 18. Time elapsed: 1:19.8s. Cache hits: 0%. Commands: 2 (cached: 0, remote: 0, local: 2) Tests finished: Pass 6. Fail 0. Fatal 0. Skip 0. Build failure 0 # local reproduce ### cmf P887605070 ### igctr P892987433 ### mai P893109069 ### icvr P893075846 ### oc P893109069 Reviewed By: xuzhao9 Differential Revision: D51332067

facebook-github-bot · 2023-11-30T00:57:53Z

This pull request was exported from Phabricator. Differential Revision: D51332067

…h#114778) Summary: We construct a unified API that can be easily add pointwise ops to be batched in the post grad Test Plan: # unit test ``` buck2 test 'fbcode//mode/dev-nosan' fbcode//caffe2/test/inductor:group_batch_fusion ``` Buck UI: https://www.internalfb.com/buck2/19b3f641-782f-4f94-a953-3ff9ce2cfa7b Test UI: https://www.internalfb.com/intern/testinfra/testrun/1125900251953016 Network: Up: 67KiB Down: 32KiB (reSessionID-c2a80f26-8227-4f78-89fc-bcbda0ae8353) Jobs completed: 18. Time elapsed: 1:19.8s. Cache hits: 0%. Commands: 2 (cached: 0, remote: 0, local: 2) Tests finished: Pass 6. Fail 0. Fatal 0. Skip 0. Build failure 0 # local reproduce ### cmf P887605070 ### igctr P892987433 ### mai P893109069 ### icvr P893075846 ### oc P893109069 Reviewed By: xuzhao9 Differential Revision: D51332067

facebook-github-bot · 2023-11-30T03:30:27Z

This pull request was exported from Phabricator. Differential Revision: D51332067

…h#114778) Summary: We construct a unified API that can be easily add pointwise ops to be batched in the post grad Test Plan: # unit test ``` buck2 test 'fbcode//mode/dev-nosan' fbcode//caffe2/test/inductor:group_batch_fusion ``` Buck UI: https://www.internalfb.com/buck2/19b3f641-782f-4f94-a953-3ff9ce2cfa7b Test UI: https://www.internalfb.com/intern/testinfra/testrun/1125900251953016 Network: Up: 67KiB Down: 32KiB (reSessionID-c2a80f26-8227-4f78-89fc-bcbda0ae8353) Jobs completed: 18. Time elapsed: 1:19.8s. Cache hits: 0%. Commands: 2 (cached: 0, remote: 0, local: 2) Tests finished: Pass 6. Fail 0. Fatal 0. Skip 0. Build failure 0 # local reproduce ### cmf P887605070 ### igctr P892987433 ### mai P893109069 ### icvr P893075846 ### oc P893109069 Reviewed By: xuzhao9 Differential Revision: D51332067

facebook-github-bot · 2023-11-30T19:21:14Z

This pull request was exported from Phabricator. Differential Revision: D51332067

…h#114778) Summary: We construct a unified API that can be easily add pointwise ops to be batched in the post grad Test Plan: # unit test ``` buck2 test 'fbcode//mode/dev-nosan' fbcode//caffe2/test/inductor:group_batch_fusion ``` Buck UI: https://www.internalfb.com/buck2/6c5d1d31-e4d1-4865-bf79-1e7ac3b6e051 Test UI: https://www.internalfb.com/intern/testinfra/testrun/1970325050015770 Network: Up: 72KiB Down: 22KiB (reSessionID-44adc8b2-54e9-453a-bd20-710cefefaed1) Jobs completed: 20. Time elapsed: 1:44.6s. Cache hits: 0%. Commands: 2 (cached: 0, remote: 0, local: 2) Tests finished: Pass 7. Fail 0. Fatal 0. Skip 0. Build failure 0 # local reproduce ### cmf P887605070 ### igctr P892987433 ### mai P893109069 ### icvr P893075846 ### oc P893109069 Reviewed By: xuzhao9 Differential Revision: D51332067

facebook-github-bot · 2023-12-04T07:35:31Z

This pull request was exported from Phabricator. Differential Revision: D51332067

yanboliang · 2023-12-06T01:01:41Z

From the error log:

2023-12-04T20:22:19.8027920Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 320, in compile_fx_inner
2023-12-04T20:22:19.8028807Z     compiled_graph = fx_codegen_and_compile(
2023-12-04T20:22:19.8029821Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 514, in fx_codegen_and_compile
2023-12-04T20:22:19.8030747Z     post_grad_passes(gm, is_inference=is_inference)
2023-12-04T20:22:19.8031788Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/fx_passes/post_grad.py", line 86, in post_grad_passes
2023-12-04T20:22:19.8032797Z     group_batch_fusion_passes(gm.graph, pre_grad=False)
2023-12-04T20:22:19.8033946Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/fx_passes/group_batch_fusion.py", line 885, in group_batch_fusion_passes
2023-12-04T20:22:19.8034941Z     apply_group_batch_fusion(graph, rule)
2023-12-04T20:22:19.8036033Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/fx_passes/group_batch_fusion.py", line 810, in apply_group_batch_fusion
2023-12-04T20:22:19.8037048Z     for subset in find_independent_subset_greedy(
2023-12-04T20:22:19.8038200Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/fx_passes/group_batch_fusion.py", line 743, in find_independent_subset_greedy
2023-12-04T20:22:19.8039203Z     find_dependent_nodes(node, node)
2023-12-04T20:22:19.8040251Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/fx_passes/group_batch_fusion.py", line 730, in find_dependent_nodes
2023-12-04T20:22:19.8041238Z     find_dependent_nodes(src_node, input_node)
2023-12-04T20:22:19.8042334Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/fx_passes/group_batch_fusion.py", line 730, in find_dependent_nodes
2023-12-04T20:22:19.8043319Z     find_dependent_nodes(src_node, input_node)
2023-12-04T20:22:19.8044396Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/fx_passes/group_batch_fusion.py", line 730, in find_dependent_nodes
2023-12-04T20:22:19.8045375Z     find_dependent_nodes(src_node, input_node)
2023-12-04T20:22:19.8045855Z   [Previous line repeated 1937 more times]
2023-12-04T20:22:19.8047030Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/fx_passes/group_batch_fusion.py", line 724, in find_dependent_nodes
2023-12-04T20:22:19.8048005Z     for input_node in cur_node.all_input_nodes:
2023-12-04T20:22:19.8048905Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/node.py", line 352, in all_input_nodes
2023-12-04T20:22:19.8049704Z     return list(self._input_nodes.keys())
2023-12-04T20:22:19.8050349Z torch._dynamo.exc.BackendCompilerFailed: backend='inductor' raised:
2023-12-04T20:22:19.8051135Z RecursionError: maximum recursion depth exceeded while calling a Python object

It seems you modify fx graph and make the input args point to the node itself, you can check this by:

python benchmarks/dynamo/huggingface.py -dcuda --inductor --training --accuracy --only MobileBertForMaskedLM

yanboliang · 2023-12-06T01:02:03Z

BTW, the PR and diff is out of sync.

…h#114778) Summary: We construct a unified API that can be easily add pointwise ops to be batched in the post grad Test Plan: # unit test ``` buck2 test 'fbcode//mode/dev-nosan' fbcode//caffe2/test/inductor:group_batch_fusion ``` Buck UI: https://www.internalfb.com/buck2/6c5d1d31-e4d1-4865-bf79-1e7ac3b6e051 Test UI: https://www.internalfb.com/intern/testinfra/testrun/1970325050015770 Network: Up: 72KiB Down: 22KiB (reSessionID-44adc8b2-54e9-453a-bd20-710cefefaed1) Jobs completed: 20. Time elapsed: 1:44.6s. Cache hits: 0%. Commands: 2 (cached: 0, remote: 0, local: 2) Tests finished: Pass 7. Fail 0. Fatal 0. Skip 0. Build failure 0 # local reproduce ### cmf P887605070 ### igctr P892987433 ### mai P893109069 ### icvr P893075846 ### oc P893109069 Reviewed By: xuzhao9 Differential Revision: D51332067

facebook-github-bot · 2023-12-07T04:13:45Z

This pull request was exported from Phabricator. Differential Revision: D51332067

…h#114778) Summary: We construct a unified API that can be easily add pointwise ops to be batched in the post grad Test Plan: # unit test ``` buck2 test 'fbcode//mode/dev-nosan' fbcode//caffe2/test/inductor:group_batch_fusion ``` Buck UI: https://www.internalfb.com/buck2/6c5d1d31-e4d1-4865-bf79-1e7ac3b6e051 Test UI: https://www.internalfb.com/intern/testinfra/testrun/1970325050015770 Network: Up: 72KiB Down: 22KiB (reSessionID-44adc8b2-54e9-453a-bd20-710cefefaed1) Jobs completed: 20. Time elapsed: 1:44.6s. Cache hits: 0%. Commands: 2 (cached: 0, remote: 0, local: 2) Tests finished: Pass 7. Fail 0. Fatal 0. Skip 0. Build failure 0 # local reproduce ### cmf P887605070 ### igctr P892987433 ### mai P893109069 ### icvr P893075846 ### oc P893109069 ### mixed precison training P898569125 # e2e test baseline f509792379 proposal f509792025 https://pxl.cl/3Xbcf Reviewed By: xuzhao9 Differential Revision: D51332067

facebook-github-bot · 2023-12-07T05:37:33Z

This pull request was exported from Phabricator. Differential Revision: D51332067

…h#114778) Summary: We construct a unified API that can be easily add pointwise ops to be batched in the post grad Test Plan: # unit test ``` buck2 test 'fbcode//mode/dev-nosan' fbcode//caffe2/test/inductor:group_batch_fusion ``` Buck UI: https://www.internalfb.com/buck2/6c5d1d31-e4d1-4865-bf79-1e7ac3b6e051 Test UI: https://www.internalfb.com/intern/testinfra/testrun/1970325050015770 Network: Up: 72KiB Down: 22KiB (reSessionID-44adc8b2-54e9-453a-bd20-710cefefaed1) Jobs completed: 20. Time elapsed: 1:44.6s. Cache hits: 0%. Commands: 2 (cached: 0, remote: 0, local: 2) Tests finished: Pass 7. Fail 0. Fatal 0. Skip 0. Build failure 0 # local reproduce ### cmf P887605070 ### igctr P892987433 ### mai P893109069 ### icvr P893075846 ### oc P893109069 ### mixed precison training P898569125 # e2e test baseline f509792379 proposal f509792025 https://pxl.cl/3Xbcf Reviewed By: xuzhao9 Differential Revision: D51332067

facebook-github-bot · 2023-12-07T06:38:07Z

This pull request was exported from Phabricator. Differential Revision: D51332067

…h#114778) Summary: We construct a unified API that can be easily add pointwise ops to be batched in the post grad Test Plan: # unit test ``` buck2 test 'fbcode//mode/dev-nosan' fbcode//caffe2/test/inductor:group_batch_fusion ``` Buck UI: https://www.internalfb.com/buck2/6c5d1d31-e4d1-4865-bf79-1e7ac3b6e051 Test UI: https://www.internalfb.com/intern/testinfra/testrun/1970325050015770 Network: Up: 72KiB Down: 22KiB (reSessionID-44adc8b2-54e9-453a-bd20-710cefefaed1) Jobs completed: 20. Time elapsed: 1:44.6s. Cache hits: 0%. Commands: 2 (cached: 0, remote: 0, local: 2) Tests finished: Pass 7. Fail 0. Fatal 0. Skip 0. Build failure 0 # local reproduce ### cmf P887605070 ### igctr P892987433 ### mai P893109069 ### icvr P893075846 ### oc P893109069 ### mixed precison training P898569125 # e2e test baseline f509792379 proposal f509792025 https://pxl.cl/3Xbcf Reviewed By: xuzhao9 Differential Revision: D51332067

facebook-github-bot · 2023-12-07T09:04:35Z

This pull request was exported from Phabricator. Differential Revision: D51332067

facebook-github-bot · 2023-12-07T09:04:43Z

This pull request was exported from Phabricator. Differential Revision: D51332067

facebook-github-bot · 2023-12-07T19:02:06Z

@pytorchbot merge

(Initiating merge automatically since Phabricator Diff has merged)

pytorchmergebot · 2023-12-07T19:03:49Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

…h#114778) Summary: We construct a unified API that can be easily add pointwise ops to be batched in the post grad Test Plan: # unit test ``` buck2 test 'fbcode//mode/dev-nosan' fbcode//caffe2/test/inductor:group_batch_fusion ``` Buck UI: https://www.internalfb.com/buck2/19b3f641-782f-4f94-a953-3ff9ce2cfa7b Test UI: https://www.internalfb.com/intern/testinfra/testrun/1125900251953016 Network: Up: 67KiB Down: 32KiB (reSessionID-c2a80f26-8227-4f78-89fc-bcbda0ae8353) Jobs completed: 18. Time elapsed: 1:19.8s. Cache hits: 0%. Commands: 2 (cached: 0, remote: 0, local: 2) Tests finished: Pass 6. Fail 0. Fatal 0. Skip 0. Build failure 0 # local reproduce ### cmf P881792289 ### igctr ### dsnn ### icvr Reviewed By: xuzhao9 Differential Revision: D51332067 Pull Request resolved: pytorch#114778 Approved by: https://github.com/xuzhao9

facebook-github-bot added the fb-exported label Nov 29, 2023

github-actions bot added module: inductor ciflow/inductor and removed fb-exported labels Nov 29, 2023

xuzhao9 self-requested a review November 29, 2023 19:38

xuzhao9 approved these changes Nov 29, 2023

View reviewed changes

mengluy0125 force-pushed the export-D51332067 branch from a014a7c to adb2046 Compare November 29, 2023 22:55

facebook-github-bot added the fb-exported label Nov 29, 2023

mengluy0125 force-pushed the export-D51332067 branch from adb2046 to 1adc1a0 Compare November 30, 2023 00:57

xuzhao9 mentioned this pull request Nov 30, 2023

[inductor] post_grad batched linear fusion #112504

Closed

mengluy0125 force-pushed the export-D51332067 branch from 1adc1a0 to f662602 Compare November 30, 2023 03:30

mengluy0125 force-pushed the export-D51332067 branch from f662602 to be3b9d3 Compare November 30, 2023 19:21

mengluy0125 force-pushed the export-D51332067 branch from be3b9d3 to 33eccc7 Compare December 4, 2023 07:35

mengluy0125 force-pushed the export-D51332067 branch from 33eccc7 to 0b4c7fa Compare December 7, 2023 04:13

mengluy0125 force-pushed the export-D51332067 branch from 0b4c7fa to 5d7ecfd Compare December 7, 2023 05:37

mengluy0125 force-pushed the export-D51332067 branch from 5d7ecfd to 4228fe8 Compare December 7, 2023 06:37

mengluy0125 force-pushed the export-D51332067 branch from 4228fe8 to acfeca2 Compare December 7, 2023 09:04

mengluy0125 force-pushed the export-D51332067 branch from acfeca2 to 1cda761 Compare December 7, 2023 09:04

pytorch-bot bot added the ciflow/trunk Trigger trunk jobs on your pull request label Dec 7, 2023

pytorchmergebot added the merging label Dec 7, 2023

pytorchmergebot added the Merged label Dec 7, 2023

pytorchmergebot removed the merging label Dec 7, 2023

pytorchmergebot closed this in b0a9641 Dec 7, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Inductor][fx pass] Fuse pointwise operators in the post grad #114778

[Inductor][fx pass] Fuse pointwise operators in the post grad #114778

mengluy0125 commented Nov 29, 2023 •

edited by pytorch-bot bot

pytorch-bot bot commented Nov 29, 2023 •

edited

facebook-github-bot commented Nov 29, 2023

github-actions bot commented Nov 29, 2023

facebook-github-bot commented Nov 29, 2023

facebook-github-bot commented Nov 30, 2023

facebook-github-bot commented Nov 30, 2023

facebook-github-bot commented Nov 30, 2023

facebook-github-bot commented Dec 4, 2023

yanboliang commented Dec 6, 2023

yanboliang commented Dec 6, 2023

facebook-github-bot commented Dec 7, 2023

facebook-github-bot commented Dec 7, 2023

facebook-github-bot commented Dec 7, 2023

facebook-github-bot commented Dec 7, 2023

facebook-github-bot commented Dec 7, 2023

facebook-github-bot commented Dec 7, 2023

pytorchmergebot commented Dec 7, 2023

[Inductor][fx pass] Fuse pointwise operators in the post grad #114778

[Inductor][fx pass] Fuse pointwise operators in the post grad #114778

Conversation

mengluy0125 commented Nov 29, 2023 • edited by pytorch-bot bot

unit test

local reproduce

cmf

igctr

dsnn

icvr

pytorch-bot bot commented Nov 29, 2023 • edited

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/114778

✅ You can merge normally! (2 Unrelated Failures)

facebook-github-bot commented Nov 29, 2023

github-actions bot commented Nov 29, 2023

This PR needs a release notes: label

facebook-github-bot commented Nov 29, 2023

facebook-github-bot commented Nov 30, 2023

facebook-github-bot commented Nov 30, 2023

facebook-github-bot commented Nov 30, 2023

facebook-github-bot commented Dec 4, 2023

yanboliang commented Dec 6, 2023

yanboliang commented Dec 6, 2023

facebook-github-bot commented Dec 7, 2023

facebook-github-bot commented Dec 7, 2023

facebook-github-bot commented Dec 7, 2023

facebook-github-bot commented Dec 7, 2023

facebook-github-bot commented Dec 7, 2023

facebook-github-bot commented Dec 7, 2023

pytorchmergebot commented Dec 7, 2023

Merge started

mengluy0125 commented Nov 29, 2023 •

edited by pytorch-bot bot

pytorch-bot bot commented Nov 29, 2023 •

edited

This PR needs a `release notes:` label