Skip to content

Commit

Permalink
Browse files Browse the repository at this point in the history
[mypyc] Refactor: move genops related code to package mypyc.irbuild (p…
…ython#8465)

This moves various genops-related modules to `mypyc/irbuild/`. Also rename
modules to have names that make sense in the new hierarchy. Module 
implementations have only minimal changes (update imports etc.).

Update documentation and docstrings to refer to the new hierarchy.

Rename test files `genops-*` to `irbuild-*`.

Work on mypyc/mypyc#714.
  • Loading branch information
JukkaL committed Mar 1, 2020
1 parent ba12948 commit aa5c02a
Show file tree
Hide file tree
Showing 37 changed files with 75 additions and 72 deletions.
16 changes: 8 additions & 8 deletions mypyc/doc/dev-intro.md
Expand Up @@ -27,7 +27,7 @@ It has these passes:
* Type check the code using mypy and infer types for variables and expressions.
* Translate the mypy AST into a mypyc-specific intermediate representation (IR).
* The IR is defined in `mypyc.ops`.
* The translation happens in `mypyc.genops`.
* The translation happens in `mypyc.irbuild`.
* Insert checks for uses of potentially uninitialized variables (`mypyc.uninit`).
* Insert exception handling (`mypyc.exceptions`).
* Insert explicit reference count inc/dec opcodes (`mypyc.refcount`).
Expand Down Expand Up @@ -94,7 +94,7 @@ what to do to implement specific kinds of mypyc features.
### Syntactic Sugar

Syntactic sugar that doesn't need additional IR operations typically
only requires changes to `mypyc.genops`.
only requires changes to `mypyc.irbuild`.


### Testing
Expand All @@ -110,16 +110,16 @@ driver is in `mypyc.test.test_run`.

If the specifics of the generated IR of a change is important
(because, for example, you want to make sure a particular optimization
is triggering), you should add a genops test as well. Test cases are
located in `test-data/genops-*.test` and the test driver is in
`mypyc.test.test_genops`. Genops tests do a direct comparison of the
is triggering), you should add an irbuild test as well. Test cases are
located in `test-data/irbuild-*.test` and the test driver is in
`mypyc.test.test_irbuild`. IR build tests do a direct comparison of the
IR output, so try to make the test as targeted as possible so as to
capture only the important details.
(Many of our existing genops tests do not follow this advice, unfortunately!)
(Many of our existing IR build tests do not follow this advice, unfortunately!)

If you pass the `--update-data` flag to pytest, it will automatically
update the expected output of any tests to match the actual
output. This is very useful for changing or creating genops tests, but
output. This is very useful for changing or creating IR build tests, but
make sure to carefully inspect the diff!

You may also need to add some definitions to the stubs used for
Expand Down Expand Up @@ -160,7 +160,7 @@ Here are some hints about how to add support for a new primitive type
Make sure all the attributes are set correctly and also define
`<foo>_rprimitive` and `is_<foo>_rprimitive`.

* Update `mypyc.genops.Mapper.type_to_rtype()`.
* Update `mypyc.irbuild.mapper.Mapper.type_to_rtype()`.

* Update `emit_box` in `mypyc.emit`.

Expand Down
8 changes: 4 additions & 4 deletions mypyc/emitmodule.py
Expand Up @@ -19,9 +19,9 @@
from mypy.fscache import FileSystemCache
from mypy.util import hash_digest

from mypyc import genopsmain
from mypyc.genopsprepare import load_type_map
from mypyc.genopsmapper import Mapper
from mypyc.irbuild.main import build_ir
from mypyc.irbuild.prepare import load_type_map
from mypyc.irbuild.mapper import Mapper
from mypyc.common import (
PREFIX, TOP_LEVEL_NAME, INT_PREFIX, MODULE_PREFIX, shared_lib_name,
)
Expand Down Expand Up @@ -202,7 +202,7 @@ def compile_scc_to_ir(
print("Compiling {}".format(", ".join(x.name for x in scc)))

# Generate basic IR, with missing exception and refcount handling.
modules = genopsmain.build_ir(
modules = build_ir(
scc, result.graph, result.types, mapper, compiler_options, errors
)
if errors.num_errors > 0:
Expand Down
Empty file added mypyc/irbuild/__init__.py
Empty file.
20 changes: 11 additions & 9 deletions mypyc/genops.py → mypyc/irbuild/builder.py
Expand Up @@ -3,12 +3,12 @@
The IRBuilder class maintains transformation state and provides access
to various helpers used to implement the transform.
The top-level transform control logic is in mypyc.genopsmain.
The top-level transform control logic is in mypyc.irbuild.main.
mypyc.genopsvisitor.IRBuilderVisitor is used to dispatch based on mypy
mypyc.irbuild.visitor.IRBuilderVisitor is used to dispatch based on mypy
AST node type to code that actually does the bulk of the work. For
example, expressions are transformed in mypyc.genexpr and functions are
transformed in mypyc.genfunc.
example, expressions are transformed in mypyc.irbuild.expression and
functions are transformed in mypyc.irbuild.function.
"""

from typing import Callable, Dict, List, Tuple, Optional, Union, Sequence, Set, Any
Expand Down Expand Up @@ -46,16 +46,18 @@
from mypyc.ops_misc import (
true_op, false_op, iter_op, next_op, py_setattr_op, import_op, get_module_dict_op
)
from mypyc.genops_for import ForGenerator, ForRange, ForList, ForIterable, ForEnumerate, ForZip
from mypyc.crash import catch_errors
from mypyc.options import CompilerOptions
from mypyc.errors import Errors
from mypyc.nonlocalcontrol import (
from mypyc.irbuild.for_helpers import (
ForGenerator, ForRange, ForList, ForIterable, ForEnumerate, ForZip
)
from mypyc.irbuild.nonlocalcontrol import (
NonlocalControl, BaseNonlocalControl, LoopNonlocalControl, GeneratorNonlocalControl
)
from mypyc.genopscontext import FuncInfo, ImplicitClass
from mypyc.genopsmapper import Mapper
from mypyc.ir_builder import LowLevelIRBuilder
from mypyc.irbuild.context import FuncInfo, ImplicitClass
from mypyc.irbuild.mapper import Mapper
from mypyc.irbuild.ll_builder import LowLevelIRBuilder

GenFunc = Callable[[], None]

Expand Down
8 changes: 4 additions & 4 deletions mypyc/genclass.py → mypyc/irbuild/classdef.py
Expand Up @@ -17,12 +17,12 @@
)
from mypyc.ops_dict import dict_set_item_op, new_dict_op
from mypyc.ops_tuple import new_tuple_op
from mypyc.genopsutil import (
from mypyc.common import SELF_NAME
from mypyc.irbuild.util import (
is_dataclass_decorator, get_func_def, is_dataclass, is_constant, add_self_to_env
)
from mypyc.genfunc import transform_method
from mypyc.common import SELF_NAME
from mypyc.genops import IRBuilder
from mypyc.irbuild.builder import IRBuilder
from mypyc.irbuild.function import transform_method


def transform_class_def(builder: IRBuilder, cdef: ClassDef) -> None:
Expand Down
File renamed without changes.
7 changes: 4 additions & 3 deletions mypyc/genexpr.py → mypyc/irbuild/expression.py
@@ -1,6 +1,7 @@
"""Transform mypy expression ASTs to mypyc IR (Intermediate Representation).
The top-level AST transformation logic is implemented in mypyc.genops.
The top-level AST transformation logic is implemented in mypyc.irbuild.visitor
and mypyc.irbuild.builder.
"""

from typing import List, Optional, Union
Expand All @@ -23,8 +24,8 @@
from mypyc.ops_tuple import list_tuple_op
from mypyc.ops_dict import new_dict_op, dict_set_item_op
from mypyc.ops_set import new_set_op, set_add_op, set_update_op
from mypyc.specialize import specializers
from mypyc.genops import IRBuilder
from mypyc.irbuild.specialize import specializers
from mypyc.irbuild.builder import IRBuilder


# Name and attribute references
Expand Down
4 changes: 2 additions & 2 deletions mypyc/genops_for.py → mypyc/irbuild/for_helpers.py
Expand Up @@ -19,14 +19,14 @@
from mypyc.ops_exc import no_err_occurred_op

if TYPE_CHECKING:
import mypyc.genops
import mypyc.irbuild.builder


class ForGenerator:
"""Abstract base class for generating for loops."""

def __init__(self,
builder: 'mypyc.genops.IRBuilder',
builder: 'mypyc.irbuild.builder.IRBuilder',
index: Lvalue,
body_block: BasicBlock,
loop_exit: BasicBlock,
Expand Down
8 changes: 4 additions & 4 deletions mypyc/genfunc.py → mypyc/irbuild/function.py
Expand Up @@ -27,10 +27,10 @@
SELF_NAME, ENV_ATTR_NAME, NEXT_LABEL_ATTR_NAME, LAMBDA_NAME, decorator_helper_name
)
from mypyc.sametype import is_same_method_signature
from mypyc.genopsutil import concrete_arg_kind, is_constant, add_self_to_env
from mypyc.genopscontext import FuncInfo, GeneratorClass, ImplicitClass
from mypyc.genstatement import transform_try_except
from mypyc.genops import IRBuilder
from mypyc.irbuild.util import concrete_arg_kind, is_constant, add_self_to_env
from mypyc.irbuild.context import FuncInfo, GeneratorClass, ImplicitClass
from mypyc.irbuild.statement import transform_try_except
from mypyc.irbuild.builder import IRBuilder


# Top-level transform functions
Expand Down
2 changes: 1 addition & 1 deletion mypyc/ir_builder.py → mypyc/irbuild/ll_builder.py
Expand Up @@ -44,7 +44,7 @@
from mypyc.rt_subtype import is_runtime_subtype
from mypyc.subtype import is_subtype
from mypyc.sametype import is_same_type
from mypyc.genopsmapper import Mapper
from mypyc.irbuild.mapper import Mapper


DictEntry = Tuple[Optional[Value], Value]
Expand Down
14 changes: 7 additions & 7 deletions mypyc/genopsmain.py → mypyc/irbuild/main.py
Expand Up @@ -16,7 +16,7 @@ def f(x: int) -> int:
The IR is implemented in mypyc.ops.
For the core of the implementation, look at build_ir() below,
mypyc.genops, and mypyc.genopsvisitor.
mypyc.irbuild.builder, and mypyc.irbuild.visitor.
"""

from collections import OrderedDict
Expand All @@ -29,13 +29,13 @@ def f(x: int) -> int:

from mypyc.errors import Errors
from mypyc.options import CompilerOptions
from mypyc.prebuildvisitor import PreBuildVisitor
from mypyc.genopsvtable import compute_vtable
from mypyc.genopsprepare import build_type_map
from mypyc.genops import IRBuilder
from mypyc.genopsvisitor import IRBuilderVisitor
from mypyc.ops import ModuleIR, ModuleIRs
from mypyc.genopsmapper import Mapper
from mypyc.prebuildvisitor import PreBuildVisitor
from mypyc.irbuild.vtable import compute_vtable
from mypyc.irbuild.prepare import build_type_map
from mypyc.irbuild.builder import IRBuilder
from mypyc.irbuild.visitor import IRBuilderVisitor
from mypyc.irbuild.mapper import Mapper


# The stubs for callable contextmanagers are busted so cast it to the
Expand Down
File renamed without changes.
Expand Up @@ -9,7 +9,7 @@
from mypyc.ops_exc import set_stop_iteration_value, restore_exc_info_op

if TYPE_CHECKING:
from mypyc.genops import IRBuilder
from mypyc.irbuild.builder import IRBuilder


class NonlocalControl:
Expand Down
4 changes: 2 additions & 2 deletions mypyc/genopsprepare.py → mypyc/irbuild/prepare.py
Expand Up @@ -12,8 +12,8 @@
DeserMaps, FUNC_NORMAL, FUNC_STATICMETHOD, FUNC_CLASSMETHOD
)
from mypyc.common import PROPSET_PREFIX
from mypyc.genopsmapper import Mapper
from mypyc.genopsutil import (
from mypyc.irbuild.mapper import Mapper
from mypyc.irbuild.util import (
get_func_def, is_dataclass, is_trait, is_extension_class, get_mypyc_attrs
)
from mypyc.errors import Errors
Expand Down
2 changes: 1 addition & 1 deletion mypyc/specialize.py → mypyc/irbuild/specialize.py
Expand Up @@ -22,7 +22,7 @@
str_rprimitive, list_rprimitive, dict_rprimitive, set_rprimitive, bool_rprimitive
)
from mypyc.ops_misc import true_op, false_op
from mypyc.genops import IRBuilder
from mypyc.irbuild.builder import IRBuilder


# Specializers are attempted before compiling the arguments to the
Expand Down
4 changes: 2 additions & 2 deletions mypyc/genstatement.py → mypyc/irbuild/statement.py
Expand Up @@ -16,10 +16,10 @@
raise_exception_op, reraise_exception_op, error_catch_op, exc_matches_op, restore_exc_info_op,
get_exc_value_op, keep_propagating_op, get_exc_info_op
)
from mypyc.nonlocalcontrol import (
from mypyc.irbuild.nonlocalcontrol import (
ExceptNonlocalControl, FinallyNonlocalControl, TryFinallyNonlocalControl
)
from mypyc.genops import IRBuilder
from mypyc.irbuild.builder import IRBuilder

GenFunc = Callable[[], None]

Expand Down
File renamed without changes.
12 changes: 6 additions & 6 deletions mypyc/genopsvisitor.py → mypyc/irbuild/visitor.py
@@ -1,6 +1,6 @@
"""Dispatcher used when transforming a mypy AST to the IR form.
mypyc.genops and mypyc.genopsmain are closely related.
mypyc.irbuild.builder and mypyc.irbuild.main are closely related.
"""

from typing_extensions import NoReturn
Expand All @@ -20,9 +20,9 @@
)

from mypyc.ops import Value
from mypyc.genops import IRVisitor, IRBuilder, UnsupportedException
from mypyc.genclass import transform_class_def
from mypyc.genfunc import (
from mypyc.irbuild.builder import IRVisitor, IRBuilder, UnsupportedException
from mypyc.irbuild.classdef import transform_class_def
from mypyc.irbuild.function import (
transform_func_def,
transform_overloaded_func_def,
transform_decorator,
Expand All @@ -31,7 +31,7 @@
transform_yield_from_expr,
transform_await_expr,
)
from mypyc.genstatement import (
from mypyc.irbuild.statement import (
transform_block,
transform_expression_stmt,
transform_return_stmt,
Expand All @@ -48,7 +48,7 @@
transform_assert_stmt,
transform_del_stmt,
)
from mypyc.genexpr import (
from mypyc.irbuild.expression import (
transform_name_expr,
transform_member_expr,
transform_super_expr,
Expand Down
File renamed without changes.
2 changes: 1 addition & 1 deletion mypyc/ops.py
Expand Up @@ -1023,7 +1023,7 @@ class PrimitiveOp(RegisterOp):
The details of the operation are defined by the 'desc'
attribute. The mypyc.ops_* modules define the supported
operations. mypyc.genops uses the descriptions to look for suitable
operations. mypyc.irbuild uses the descriptions to look for suitable
primitive ops.
"""

Expand Down
2 changes: 1 addition & 1 deletion mypyc/ops_int.py
Expand Up @@ -63,7 +63,7 @@ def int_compare_op(op: str, c_func_name: str) -> None:
int_binary_op('%', 'CPyTagged_Remainder', error_kind=ERR_MAGIC)

# this should work because assignment operators are parsed differently
# and the code in genops that handles it does the assignment
# and the code in irbuild that handles it does the assignment
# regardless of whether or not the operator works in place anyway
int_binary_op('+=', 'CPyTagged_Add')
int_binary_op('-=', 'CPyTagged_Subtract')
Expand Down
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
2 changes: 1 addition & 1 deletion mypyc/test/test_emitfunc.py
Expand Up @@ -12,7 +12,7 @@
ClassIR, RInstance, SetAttr, Op, Value, int_rprimitive, bool_rprimitive,
list_rprimitive, dict_rprimitive, object_rprimitive, FuncSignature,
)
from mypyc.genopsvtable import compute_vtable
from mypyc.irbuild.vtable import compute_vtable
from mypyc.emit import Emitter, EmitterContext
from mypyc.emitfunc import generate_native_function, FunctionEmitterVisitor
from mypyc.ops_primitive import binary_ops
Expand Down
26 changes: 13 additions & 13 deletions mypyc/test/test_genops.py → mypyc/test/test_irbuild.py
Expand Up @@ -15,19 +15,19 @@
from mypyc.options import CompilerOptions

files = [
'genops-basic.test',
'genops-lists.test',
'genops-dict.test',
'genops-statements.test',
'genops-nested.test',
'genops-classes.test',
'genops-optional.test',
'genops-tuple.test',
'genops-any.test',
'genops-generics.test',
'genops-try.test',
'genops-set.test',
'genops-strip-asserts.test',
'irbuild-basic.test',
'irbuild-lists.test',
'irbuild-dict.test',
'irbuild-statements.test',
'irbuild-nested.test',
'irbuild-classes.test',
'irbuild-optional.test',
'irbuild-tuple.test',
'irbuild-any.test',
'irbuild-generics.test',
'irbuild-try.test',
'irbuild-set.test',
'irbuild-strip-asserts.test',
]


Expand Down
4 changes: 2 additions & 2 deletions mypyc/test/testutil.py
Expand Up @@ -14,11 +14,11 @@
from mypy.test.config import test_temp_dir
from mypy.test.helpers import assert_string_arrays_equal

from mypyc.genopsmain import build_ir
from mypyc.options import CompilerOptions
from mypyc.ops import FuncIR
from mypyc.errors import Errors
from mypyc.genopsmapper import Mapper
from mypyc.irbuild.main import build_ir
from mypyc.irbuild.mapper import Mapper
from mypyc.test.config import test_data_prefix

# The builtins stub used during icode generation test cases.
Expand Down

0 comments on commit aa5c02a

Please sign in to comment.