Skip to content
This repository has been archived by the owner on Dec 1, 2023. It is now read-only.

Commit

Permalink
Merge pull request #73 from Quansight-Labs/discrete-tasks
Browse files Browse the repository at this point in the history 
Discrete tasks
  • Loading branch information
@saulshanabrook
saulshanabrook committed Jul 27, 2019
2 parents 60f4dea + b502ccc commit b97efd0
Show file tree
Hide file tree
Showing 17 changed files with 290 additions and 330 deletions.
180 changes: 46 additions & 134 deletions Demo.ipynb
View file
Original file line number Diff line number Diff line change
@@ -1,8 +1,17 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# `metadsl`\n",
"\n",
"...is a framework for creating domain specific languages in Python."
]
},
{
"cell_type": "code",
"execution_count": 2,
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
Expand All @@ -11,229 +20,132 @@
"from metadsl import *\n",
"from metadsl_core import *\n",
"from metadsl_visualize import *\n",
"import metadsl_core.vec\n",
"set_rule(core_rules)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# `metadsl` Demo\n",
"\n",
"In this notebook we show a few examples of using `metadsl`. The outputs are interactive widgets that show the progress of replacing the expressions. You can drag the slider to move from the original expression to the final replaced one"
"set_rule(all_rules)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Indexing a vector"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can create a vector type and then index it, to see the conversion progress:"
"Here, we show a NumPy API compatible DSL that we have started writing. We can create a symbolic expression and then execute it. The widget shows the different stages of execution:"
]
},
{
"cell_type": "code",
"execution_count": 3,
"execution_count": 29,
"metadata": {},
"outputs": [
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "36999a21c45047afad94237d3d4e011e",
"model_id": "beea18dda6ef44919f9f8a8c73fa2a4e",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"VBox(children=(IntSlider(value=0, max=1), Output()))"
"VBox(children=(IntSlider(value=0, max=3), Output()))"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"Vec.create(Integer.from_int(1), Integer.from_int(2))[Integer.from_int(0)]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"If we look at how this is implemented, we see how we define a rule to replace indexing a vector:"
"N = 100\n",
"expr = (arange(N) + arange(N))[5].to_ndarray()\n",
"expr"
]
},
{
"cell_type": "code",
"execution_count": 4,
"execution_count": 30,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"\u001b[0;31mSignature:\u001b[0m \u001b[0mmetadsl_core\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mvec\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mgetitem\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mi\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;34m'int'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mxs\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;34m'typing.Sequence[T]'\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m->\u001b[0m \u001b[0;34m'R[T]'\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0;31mCall signature:\u001b[0m \u001b[0mmetadsl_core\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mvec\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mgetitem\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mexpr\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mobject\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m->\u001b[0m \u001b[0mIterable\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mmetadsl\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrules\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mReplacement\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0;31mType:\u001b[0m MatchRule\n",
"\u001b[0;31mString form:\u001b[0m metadsl_core.vec.getitem\n",
"\u001b[0;31mFile:\u001b[0m ~/p/metadsl/metadsl_core/vec.py\n",
"\u001b[0;31mSource:\u001b[0m \n",
"\u001b[0;34m@\u001b[0m\u001b[0mregister\u001b[0m \u001b[0;31m# type: ignore\u001b[0m\u001b[0;34m\u001b[0m\n",
"\u001b[0;34m\u001b[0m\u001b[0;34m@\u001b[0m\u001b[0mrule\u001b[0m\u001b[0;34m\u001b[0m\n",
"\u001b[0;34m\u001b[0m\u001b[0;32mdef\u001b[0m \u001b[0mgetitem\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mi\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mint\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mxs\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mtyping\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mSequence\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mT\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m->\u001b[0m \u001b[0mR\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mT\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\n",
"\u001b[0;34m\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0mVec\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mT\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcreate\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0mxs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mInteger\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfrom_int\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mi\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;32mlambda\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mxs\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mi\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0;31mClass docstring:\u001b[0m\n",
"Creates a replacement rule given a function that maps from wildcard inputs\n",
"to two things, a template expression tree and a replacement thunk.\n",
"\n",
"If the template matches an expression, it will be replaced with the result of the thunk, replacing\n",
"the input args with the nodes at their locations in the template.\n",
"\n",
"You can also return None from the rule to signal that it won't match.\n"
"10"
]
},
"execution_count": 30,
"metadata": {},
"output_type": "display_data"
"output_type": "execute_result"
}
],
"source": [
"metadsl_core.vec.getitem??"
"execute_core(expr)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## indexing an array and conversion"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now we can try creating some NumPy arrays and indexing them. We see that through the replacement system we figure out if we are indexing with a tuple or an integer. This is an easier place to compile to different backends (like LLVM) than just the raw NumPy calls:"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "15702498ee3c4463a06a28cb7c694c24",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"VBox(children=(IntSlider(value=0, max=28), Output()))"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"arange(10)[5]"
"Now, we can add an optimization replacement, so that we do the getitem before adding the two expressions:"
]
},
{
"cell_type": "code",
"execution_count": 6,
"execution_count": 32,
"metadata": {},
"outputs": [
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "0d7fac3bab474d52ab57a6b158829ca5",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"VBox(children=(IntSlider(value=0, max=43), Output()))"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"outputs": [],
"source": [
"arange(10)[(2,)]"
"@register\n",
"@rule\n",
"def optimize_getitem_add(l: NDArray, r: NDArray, idx: IndxType) -> R[NDArray]:\n",
" return (\n",
" # expression to match against\n",
" (l + r)[idx],\n",
" # expression to replace it with\n",
" l[idx] + r[idx]\n",
" )"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now we can show one way of compiling these calls, by replacing them with the corresponding NumPy calls, to compute the results:"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "762d9f4a08d445d998369c3960229479",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"VBox(children=(IntSlider(value=0, max=36), Output()))"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"unbox_ndarray_compat(arange(10)[5])"
"Now, when we look at the execution steps, we see that it is doing each getitem before adding, which should be more efficient:"
]
},
{
"cell_type": "code",
"execution_count": 8,
"execution_count": 34,
"metadata": {},
"outputs": [
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "341f15f1fc1147f6bb0ad9c2387f119b",
"model_id": "6bb9b1cef44a4a7c81629de04403f81b",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"VBox(children=(IntSlider(value=0, max=52), Output()))"
"VBox(children=(IntSlider(value=0, max=3), Output()))"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"unbox_ndarray_compat(arange(10)[(2,)])"
"expr = (arange(N) + arange(N))[5].to_ndarray()\n",
"expr"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In this demo, we show how we can break up the NumPy API into different layers, all of which are extensible:\n",
"This shows how we can break up the NumPy API into different layers, all of which are extensible:\n",
"\n",
"1. A compatibility layer that works like the existing NumPy API, except isn't limited to the Python types of the current API\n",
"2. A type safe version of this API. The conversion between the compatability layer and this layer is extensible, so that third party authors can add new conversion between their own Python objects and the typed representation.\n",
"3. (Not implemented yet) A mathematical representation of the array operations that generalizes the api to a much smaller subset of functions.\n",
"4. A backend layer that translates either back to Python calls or source code, or to other targets like LLVM or Tensorflow.\n",
"3. A backend layer that translates either back to Python calls or source code, or to other targets like LLVM or Tensorflow.\n",
"\n",
"The key is that all these layers are composable, so you could have different frontends for any of them or add your own. This is all done through a typed replacement system that is compatible with static analysis using MyPy.\n",
"\n",
"The key is that all these layers are composable, so you could have different frontends for any of them or add your own. This is all done through a typed replacement system that is compatible with static analysis using MyPy."
"`metadsl` is the glue that allows us to define this API in a way that enables greater greater collaboration. Since each layer has it's own well defined specification, different projects can extend and target these seperate layers, while minimizing the need to explicitly opt in to collaborate. The goal is to enable greater reuse, interoperability, and extensability within the Python scientific ecosystem."
]
}
],
Expand Down
6 changes: 3 additions & 3 deletions docs/Concepts.ipynb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -151,7 +151,7 @@
},
{
"cell_type": "code",
"execution_count": 9,
"execution_count": 5,
"metadata": {},
"outputs": [
{
Expand All @@ -177,7 +177,7 @@
},
{
"cell_type": "code",
"execution_count": 10,
"execution_count": 6,
"metadata": {},
"outputs": [
{
Expand All @@ -186,7 +186,7 @@
"246"
]
},
"execution_count": 10,
"execution_count": 6,
"metadata": {},
"output_type": "execute_result"
}
Expand Down

0 comments on commit b97efd0

Please sign in to comment.