Skip to content
This repository has been archived by the owner on Dec 1, 2023. It is now read-only.

Commit

Permalink
Merge pull request #57 from Quansight-Labs/consolidate
Browse files Browse the repository at this point in the history 
Simplify Implementation and Inspect Types
  • Loading branch information
@saulshanabrook
saulshanabrook committed May 14, 2019
2 parents 0adafa4 + 2d7a673 commit 683078d
Show file tree
Hide file tree
Showing 50 changed files with 3,156 additions and 1,584 deletions.
2 changes: 0 additions & 2 deletions .coveragerc
View file
Original file line number Diff line number Diff line change
Expand Up @@ -5,5 +5,3 @@ source =
[report]
omit =
*_test.py
**/test_*.py
metadsl/*/**
17 changes: 16 additions & 1 deletion README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -37,15 +37,30 @@ flit install --symlink
### Tests

This runs mypy and tests, and reports coverage.

```bash
pytest
pytest --cov
# open coverage file
open htmlcov/index.html
```

You can also test that the documentation notebooks run correctly, but this
[must be run separately from the code coverage](https://github.com/computationalmodelling/nbval/issues/116):

```bash
pytest docs/*.ipynb --nbval
````


### Docs

```bash
sphinx-autobuild docs docs/_build/html/
```


---

Switch to Expression syntax. everything is expression with _function and _arguments

Use generic types for generic types, only allow these options on expression
191 changes: 42 additions & 149 deletions docs/Concepts.ipynb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -18,9 +18,9 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"## Instances and Calls\n",
"## Expression\n",
"\n",
"The two building blocks we start with are, `Call`s and `Instance`s:"
"We start with a building block `Expression`s:"
]
},
{
Expand All @@ -29,22 +29,23 @@
"metadata": {},
"outputs": [],
"source": [
"from __future__ import annotations\n",
"\n",
"import dataclasses\n",
"import metadsl\n",
"\n",
"@dataclasses.dataclass\n",
"class MyObject(metadsl.Instance):\n",
" @metadsl.call(lambda self: MyObject)\n",
" def do_things(self) -> \"MyObject\":\n",
"class MyObject(metadsl.Expression):\n",
" @metadsl.expression\n",
" def do_things(self) -> MyObject:\n",
" ...\n",
"\n",
" @metadsl.call(lambda self, other: MyObject)\n",
" def __add__(self, other: \"MyObject\") -> \"MyObject\":\n",
" @metadsl.expression\n",
" def __add__(self, other: MyObject) -> MyObject:\n",
" ...\n",
"\n",
" \n",
"@metadsl.call(lambda x: MyObject)\n",
"def create_object(x: int) -> MyObject:\n",
"@metadsl.expression\n",
"def create_object(x: metadsl.E[int]) -> MyObject:\n",
" ..."
]
},
Expand All @@ -56,7 +57,7 @@
{
"data": {
"text/plain": [
"MyObject(_call=Call(create_object, (123,)))"
"MyObject(create_object, (123,))"
]
},
"execution_count": 2,
Expand All @@ -77,7 +78,7 @@
{
"data": {
"text/plain": [
"MyObject(_call=Call(__add__, (MyObject(_call=Call(do_things, (MyObject(_call=Call(create_object, (123,))),))), MyObject(_call=Call(create_object, (123,))))))"
"MyObject(MyObject.__add__, (MyObject(MyObject.do_things, (MyObject(create_object, (123,)),)), MyObject(create_object, (123,))))"
]
},
"execution_count": 3,
Expand All @@ -95,115 +96,10 @@
"source": [
"It is useful to keep in mind the strict typing constraints here, not all of which can be faithfully checked by MyPy:\n",
"\n",
"1. The arguments in a `Call` should fit the signature of the function in the call.\n",
"2. The call within the instance should have represent a function whose return type is the type of the instance.\n",
"3. The `type_fn` is the first argument passed into the `call` decorator. It should take in the same signature as the function itself, but return a function that maps from a `Call` to the return value.\n",
"\n",
"Let's check the first two of these for `o`. We see that the call's function is `create_object`, which takes in an `int` and returns a `MyObject`. It's argument is indeed an `int`, so the first is true. And the instances holding it is of type `MyObject`, which is its return type.\n",
"\n",
"For the third, we see that for both calls, it is simply creating a new `MyObject` with the call."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Expressions and Recursive Calls\n",
"\n",
"You might notice that this representation is a bit verbose. It actually contains unneccesary information, which is the type that each function returns. We don't need to store this,\n",
"because we can always recreate it given the type functions we have specfifed.\n",
"\n",
"We can translate these instances into `RecursiveCall`s, which have the same structure as `Call` object but different typing constraints:"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"RecursiveCall(__add__, (RecursiveCall(do_things, (RecursiveCall(create_object, (123,)),)), RecursiveCall(create_object, (123,))))"
]
},
"execution_count": 4,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"r = metadsl.to_expression(o.do_things() + o)\n",
"r"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"__add__(do_things(create_object(123)), create_object(123))\n"
]
}
],
"source": [
"print(r)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The relationships here are:\n",
"1. The result of `to_expression` is a `RecursiveCall` if the argument is a `Instance` and the original object otherwise. \n",
"2. The args in a `RecursiveCall` should either be the right type for the function, if they are not an `Instance`, or a `RecursiveCall` that will return that `Instance` subtype, if they are."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can always get back to original nested instance version of the call:"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"MyObject(_call=Call(__add__, (MyObject(_call=Call(do_things, (MyObject(_call=Call(create_object, (123,))),))), MyObject(_call=Call(create_object, (123,))))))"
]
},
"execution_count": 6,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"metadsl.from_expression(r)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This will recursively transform expression recursively. This leads us to another relationship:\n",
"1. The arguments in a `Expression` should fit the signature of the `_function`.\n",
"2. The return type of the `_function` should correspond to the type of the `Expression`.\n",
"\n",
"1. `from_expression(to_expression(x)) == x` for all `x`, if it is in `Instance` or any other Python object."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Why do we have these two forms? Well we need the `Instance` form, to provide a nice typed Python API. You should always be dealing in that form when you are calling the Python functions, so that the typing is enforced by MyPy. However, it's a bit more complicated to modify the graph in this form. So whenever we are traversing the graph in some way, we first convert it to expressions, so that we don't have to deal with the intermingled types."
"Let's check the first two of these for `o`. We see that the expression's function is `create_object`, which takes in an `metadsl.E[int]` and returns a `MyObject`. What is `metadsl.E`? It is a type alias for `Union[T, metadsl.LiteralExpression[T]]`. This represents anything that could be a python literal, or a leaf of the expression tree. The argument here is an `int`, which is compatible with the argument hint. And the instances holding it is of type `MyObject`, which is its return type."
]
},
{
Expand All @@ -217,71 +113,71 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"Often, it's helpful to think about replacements we do on the graph. By writing logic like this, we can then apply it to any nodes on the graph any number of times.\n",
"We can define a possible replacement rule for a single expression, and then have that execute repeatedly on the graph.\n",
"\n",
"We can combine a bunch of replacements and have them execute repeatedly on all nodes of the graph, until no more apply:"
"We can also combine several rule and execute them together:"
]
},
{
"cell_type": "code",
"execution_count": 8,
"execution_count": 4,
"metadata": {},
"outputs": [],
"source": [
"rules = metadsl.RulesRepeatFold()\n",
"applier = metadsl.RuleApplier(rules)\n",
"\n",
"@rules.append\n",
"@metadsl.rule(None, None)\n",
"def _add(x: int, y: int):\n",
" return create_object(x) + create_object(y), lambda: create_object(x + y)\n",
"@metadsl.rule\n",
"def _add(x: metadsl.E[int], y: metadsl.E[int]):\n",
" return create_object(x) + create_object(y), create_object(x + y) if isinstance(x, int) and isinstance(y, int) else None\n",
"\n",
"@rules.append\n",
"@metadsl.rule(None)\n",
"def _do_things(x: int):\n",
" return create_object(x).do_things(), lambda: create_object(x * 2)"
"@metadsl.rule\n",
"def _do_things(x: metadsl.E[int]):\n",
" return create_object(x).do_things(), create_object(x * 2) if isinstance(x, int) else None"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The requirements for these replacements is that they take in some arguments, which can match any expression in the graph.\n",
"Their first return value, build up template expression based on the inputs, that shows what it should match again. The second\n",
"is a thunk that returns the resulting replacement. Note that both should have the same type, because all replacements should be equivalent.\n",
"Their first return value builds up a template expression based on the inputs, that shows what it should match again. The second\n",
"is a is the resulting to replace it with. Note that both should have the same type, because if you replace an expression it should not invalidate\n",
"the type of something it is a part of:\n",
"\n",
"We can call these on an instance and it will return a replaced version of it:"
"We can call these on an expression and it will return a replaced version of it:"
]
},
{
"cell_type": "code",
"execution_count": 10,
"execution_count": 5,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"MyObject(_call=Call(__add__, (MyObject(_call=Call(create_object, (123,))), MyObject(_call=Call(create_object, (123,))))))\n",
"MyObject(_call=Call(create_object, (246,)))\n"
"MyObject.__add__(create_object(123), create_object(123))\n",
"create_object(246)\n"
]
}
],
"source": [
"print(o+o)\n",
"print(applier(o + o))"
"print(rules(o + o))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Just like we have a function that creates a `MyObject` from an `int`, we can have a similar one that does the opposite:"
"We can also have a rule that unwraps the `int` from the object:"
]
},
{
"cell_type": "code",
"execution_count": 12,
"execution_count": 6,
"metadata": {},
"outputs": [
{
Expand All @@ -290,32 +186,29 @@
"246"
]
},
"execution_count": 12,
"execution_count": 6,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"@metadsl.call(lambda o: applier)\n",
"def unwrap_object(o: MyObject) -> int:\n",
"@metadsl.expression\n",
"def unwrap_object(o: MyObject) -> metadsl.E[int]:\n",
" ...\n",
" \n",
"@rules.append\n",
"@metadsl.pure_rule(None)\n",
"def _unwrap_object(i: int):\n",
"@metadsl.rule\n",
"def _unwrap_object(i: metadsl.E[int]):\n",
" return unwrap_object(create_object(i)), i\n",
"\n",
"unwrap_object(o + o)"
"rules(unwrap_object(o + o))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This is nice, because now we can write our unboxing as a replacement, which means it's nice and type safe.\n",
"\n",
"You notice that here we are returning `replacements` from the `type_fn`. This means that `replacements` will be called with the created `Call(unwrap_object, (o))` object, which will in turn call all the replacements on it, \n",
"including that which we defined below for the unwrapping. So this *should* return an `int`, so it does violate the typing constraints we first wrote out about `call`s."
"This is nice, because now we can write our unboxing as a replacement, which means it's nice and type safe."
]
}
],
Expand Down

0 comments on commit 683078d

Please sign in to comment.