Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix maximum recursion depth exceeded #860

Closed
wants to merge 1 commit into from

Conversation

hlky
Copy link
Contributor

@hlky hlky commented Jul 30, 2023

Fix for error encountered when compiling large graph. This PR simply increases the recursion limit in compiler/stable_set.py.

  File "aitemplate\compiler\compiler.py", line 260, in compile_model
    graph = compiler.transform.optimize_graph(
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "aitemplate\compiler\transform\optimize_graph.py", line 146, in optimize_graph
    sorted_graph = func(sorted_graph, workdir)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "aitemplate\compiler\transform\fuse_group_ops.py", line 762, in fuse_group_ops
    sorted_graph = fuse_group_layernorm_ops(sorted_graph)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "aitemplate\compiler\transform\fuse_group_ops.py", line 732, in fuse_group_layernorm_ops
    sorted_graph = _fuse_group_ops_by_type(sorted_graph, op_type, workdir)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "aitemplate\compiler\transform\fuse_group_ops.py", line 711, in _fuse_group_ops_by_type
    if not _group_ops_by_type(sorted_graph, op_type, workdir):
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "aitemplate\compiler\transform\fuse_group_ops.py", line 464, in _group_ops_by_type
    dependency_graph = _get_dependency_graph(sorted_graph, op_type)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "aitemplate\compiler\transform\fuse_group_ops.py", line 367, in _get_dependency_graph
    _dfs(tensor, op_type, visited)
  File "aitemplate\compiler\transform\fuse_group_ops.py", line 304, in _dfs
    descendants.update(_dfs(output, op_type, visited))
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "aitemplate\compiler\transform\fuse_group_ops.py", line 304, in _dfs
    descendants.update(_dfs(output, op_type, visited))
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "aitemplate\compiler\transform\fuse_group_ops.py", line 304, in _dfs
    descendants.update(_dfs(output, op_type, visited))
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  [Previous line repeated 979 more times]
  File "aitemplate\compiler\transform\fuse_group_ops.py", line 301, in _dfs
    for op in tensor.dst_ops():
  File "aitemplate\compiler\stable_set.py", line 70, in __iter__
    return list(self._d).__iter__()
           ^^^^^^^^^^^^^^^^^^^^^^^^
RecursionError: maximum recursion depth exceeded while calling a Python object

@facebook-github-bot facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Jul 30, 2023
@hlky
Copy link
Contributor Author

hlky commented Aug 3, 2023

Just for further context, this issue was encountered while compiling SDXL. I've had a report that this issue still occurs, I'll look into it at some point, it may require setrecursionlimit somewhere else, and the issue can be bypassed by commenting out fuse_group_layernorm_ops or the entire fuse_group_ops

This was referenced Aug 7, 2023
@hlky hlky closed this Aug 19, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants