Skip to content

Commit

Permalink
remove store_attrs
Browse files Browse the repository at this point in the history
  • Loading branch information
jph00 committed Sep 3, 2020
1 parent c5826e3 commit ea1c33f
Show file tree
Hide file tree
Showing 5 changed files with 148 additions and 199 deletions.
58 changes: 29 additions & 29 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ Here's a (somewhat) quick tour of a few higlights, showing examples from each of

All fast.ai projects, including this one, are built with [nbdev](https://nbdev.fast.ai), which is a full literate programming environment built on Jupyter Notebooks. That means that every piece of documentation, including the page you're reading now, can be accessed as interactive Jupyter notebooks. In fact, you can even grab a link directly to a notebook running interactively on Google Colab - if you want to follow along with this tour, click the link below, or click the badge at the top of the page:

```
```python
colab_link('index')
```

Expand All @@ -34,7 +34,7 @@ colab_link('index')

The full docs are available at [fastcore.fast.ai](https://fastcore.fast.ai). The code in the examples and in all fast.ai libraries follow the [fast.ai style guide](https://docs.fast.ai/dev/style.html). In order to support interactive programming, all fast.ai libraries are designed to allow for `import *` to be used safely, particular by ensuring that [`__all__`](https://riptutorial.com/python/example/2894/the---all---special-variable) is defined in all packages. In order to see where a function is from, just type it:

```
```python
coll_repr
```

Expand All @@ -61,13 +61,13 @@ fastcore's testing module is designed to work well with [nbdev](https://nbdev.fa

Tests look like this:

```
```python
test_eq(coll_repr(range(1000), 5), '(#1000) [0,1,2,3,4...]')
```

That's an example from the docs for `coll_repr`. As you see, it's not showing you the output directly. Here's what that would look like:

```
```python
coll_repr(range(1000), 5)
```

Expand All @@ -80,11 +80,11 @@ coll_repr(range(1000), 5)

So, the test is actually showing you what the output looks like, because if the function call didn't return `'(#1000) [0,1,2,3,4...]'`, then the test would have failed.

So every test shown in the docs is also showing you the behavior of the library --- and visa versa!
So every test shown in the docs is also showing you the behavior of the library --- and vice versa!

Test functions always start with `test_`, and then follow with the operation being tested. So `test_eq` tests for equality (as you saw in the example above). This includes tests for equality of arrays and tensors, lists and generators, and many more:

```
```python
test_eq([0,1,2,3], np.arange(4))
```

Expand All @@ -105,42 +105,42 @@ If you want to check that objects are the same type, rather than the just contai

You can test with any comparison function using `test`, e.g test whether an object is less than:

```
```python
test(2, 3, operator.lt)
```

You can even test that exceptions are raised:

```
```python
def divide_zero(): return 1/0
test_fail(divide_zero)
```

...and test that things are printed to stdout:

```
```python
test_stdout(lambda: print('hi'), 'hi')
```

### Foundations

fast.ai is unusual in that we often use [mixins](https://en.wikipedia.org/wiki/Mixin) in our code. Mixins are widely used in many programming languages, such as Ruby, but not so much in Python. We use mixins to attach new behavior to existing libraries, or to allow modules to add new behavior to our own classes, such as in extension modules. One useful example of a mixin we define is `Path.ls`, which lists a directory and returns an `L` (an extended list class which we'll discuss shortly):

```
```python
p = Path('images')
p.ls()
```




(#3) [Path('images/att_00000.png'),Path('images/puppy.jpg'),Path('images/mnist3.png')]
(#6) [Path('images/att_00007.png'),Path('images/att_00000.png'),Path('images/puppy.jpg'),Path('images/att_00006.png'),Path('images/mnist3.png'),Path('images/att_00005.png')]



You can easily add you own mixins with the `patch` [decorator](https://realpython.com/primer-on-python-decorators/), which takes advantage of Python 3 [function annotations](https://www.python.org/dev/peps/pep-3107/#parameters) to say what class to patch:

```
```python
@patch
def num_items(self:Path): return len(self.ls())

Expand All @@ -150,15 +150,15 @@ p.num_items()



3
6



We also use `**kwargs` frequently. In python `**kwargs` in a parameter like means "*put any additional keyword arguments into a dict called `kwargs`*". Normally, using `kwargs` makes an API quite difficult to work with, because it breaks things like tab-completion and popup lists of signatures. `utils` provides `use_kwargs` and `delegates` to avoid this problem. See our [detailed article on delegation](https://www.fast.ai/2019/08/06/delegation/) on this topic.

`GetAttr` solves a similar problem (and is also discussed in the article linked above): it's allows you to use Python's exceptionally useful `__getattr__` magic method, but avoids the problem that normally in Python tab-completion and docs break when using this. For instance, you can see here that Python's `dir` function, which is used to find the attributes of a python object, finds everything inside the `self.default` attribute here:

```
```python
class Author:
def __init__(self, name): self.name = name

Expand All @@ -179,10 +179,10 @@ p = ProductPage(Author("Jeremy"), 1.50, 0.50)

Looking at that `ProductPage` example, it's rather verbose and duplicates a lot of attribute names, which can lead to bugs later if you change them only in one place. `fastcore` provides `store_attr` to simplify this common pattern. It also provides `basic_repr` to give simple objects a useful `repr`:

```
```python
class ProductPage:
store_attrs = 'author,price,cost'
def __init__(self,author,price,cost): store_attr(self)
def __init__(self,author,price,cost): store_attr()
__repr__ = basic_repr(store_attrs)

ProductPage("Jeremy", 1.50, 0.50)
Expand All @@ -197,7 +197,7 @@ ProductPage("Jeremy", 1.50, 0.50)

One of the most interesting `fastcore` functions is the `funcs_kwargs` decorator. This allows class behavior to be modified without sub-classing. This can allow folks that aren't familiar with object-oriented progressing to customize your class more easily. Here's an example of a class that uses `funcs_kwargs`:

```
```python
@funcs_kwargs
class T:
_methods=['some_method']
Expand All @@ -223,7 +223,7 @@ Like most languages, Python allows for very concise syntax for some very common
On this basis, `fastcore` has just one type that has a single letter name:`L`. The reason for this is that it is designed to be a replacement for `list`, so we want it to be just as easy to use as `[1,2,3]`. Here's how to create that as an `L`:

```
```python
L(1,2,3)
```

Expand All @@ -236,49 +236,49 @@ L(1,2,3)

The first thing to notice is that an `L` object includes in its representation its number of elements; that's the `(#3)` in the output above. If there's more than 10 elements, it will automatically truncate the list:

```
```python
p = L.range(20).shuffle()
p
```




(#20) [3,7,14,16,1,17,19,0,4,8...]
(#20) [10,7,9,17,18,4,16,15,19,14...]



`L` contains many of the same indexing ideas that NumPy's `array` does, including indexing with a list of indexes, or a boolean mask list:

```
```python
p[2,4,6]
```




(#3) [14,1,19]
(#3) [9,18,16]



It also contains other methods used in `array`, such as `L.argwhere`:

```
```python
p.argwhere(ge(15))
```




(#5) [3,5,6,12,16]
(#5) [3,4,6,7,8]



As you can see from this example, `fastcore` also includes a number of features that make a functional style of programming easier, such as a full range of boolean functions (e.g `ge`, `gt`, etc) which give the same answer as the functions from Python's `operator` module if given two parameters, but return a [curried function](https://en.wikipedia.org/wiki/Currying) if given one parameter.

There's too much functionality to show it all here, so be sure to check the docs. Many little things are added that we thought should have been in `list` in the first place, such as making this do what you'd expect (which is an error with `list`, but works fine with `L`):

```
```python
1 + L(2,3,4)
```

Expand All @@ -293,7 +293,7 @@ There's too much functionality to show it all here, so be sure to check the docs

Most Python programmers use object oriented methods and inheritance to allow different objects to behave in different ways even when called with the same method name. Some languages use a very different approach, such as Julia, which uses [multiple dispatch generic functions](https://docs.julialang.org/en/v1/manual/methods/). Python provides [single dispatch generic functions](https://www.python.org/dev/peps/pep-0443/) as part of the standard library. `fastcore` provides multiple dispatch, with the `typedispatch` decorator (which is actually an instance of `DispatchReg`):

```
```python
@typedispatch
def f_td_test(x:numbers.Integral, y): return x+1
@typedispatch
Expand Down Expand Up @@ -322,7 +322,7 @@ This approach to dispatch is particularly useful for adding implementations of f

`Transform` looks for three special methods, <code>encodes</code>, <code>decodes</code>, and <code>setups</code>, which provide the implementation for [`__call__`](https://www.python-course.eu/python3_magic_methods.php), `decode`, and `setup` respectively. For instance:

```
```python
class A(Transform):
def encodes(self, x): return x+1

Expand All @@ -338,7 +338,7 @@ A()(1)

For simple transforms like this, you can also use `Transform` as a decorator:

```
```python
@Transform
def f(x): return x+1

Expand All @@ -354,7 +354,7 @@ f(1)

Transforms can be composed into a `Pipeline`:

```
```python
@Transform
def g(x): return x/2

Expand Down
8 changes: 1 addition & 7 deletions fastcore/transform.py
Original file line number Diff line number Diff line change
Expand Up @@ -101,15 +101,9 @@ def _call(self, fn, x, split_idx=None, **kwargs):
# Cell
class DisplayedTransform(Transform):
"A transform with a `__repr__` that shows its attrs"
store_attrs=''
def __init__(self, **kwargs):
super().__init__(**kwargs)
store_attr()

@property
def name(self):
s = f" -- {attrdict(self, *self.store_attrs.split(','))}" if self.store_attrs else ''
return super().name + s
def name(self): return f"{super().name} -- {getattr(self,'__stored_args__',{})}"

This comment has been minimized.

Copy link
@graingert

graingert Sep 3, 2020

https://docs.python.org/3/reference/lexical_analysis.html#reserved-classes-of-identifiers

Any use of __*__ names, in any context, that does not follow explicitly documented use, is subject to breakage without warning.


# Cell
class ItemTransform(Transform):
Expand Down
39 changes: 18 additions & 21 deletions fastcore/utils.py
Original file line number Diff line number Diff line change
Expand Up @@ -83,26 +83,22 @@ def __enter__(self): pass
def __exit__(self, *args): return True

# Cell
def store_attr(nms=None, self=None, but=None):
"Store params named in comma-separated `nms` from calling context into attrs in `self`"
def _store_attr(self, **attrs):
for n,v in attrs.items():
setattr(self, n, v)
self.__stored_args__[n] = v

# Cell
def store_attr(names=None, self=None, but=None, **attrs):
"Store params named in comma-separated `names` from calling context into attrs in `self`"
fr = inspect.currentframe().f_back
args,varargs,keyw,locs = inspect.getargvalues(fr)
if self is None: self = locs[args[0]]
if nms is None: nms = getattr(self, 'store_attrs', None)
if nms: ns = re.split(', *', nms)
else: ns = args[1:]
if but: ns = [o for o in ns if o not in L(but)]

while fr and ns:
args,varargs,keyw,locs = inspect.getargvalues(fr)
found = []
for n in ns:
if n in locs:
setattr(self, n, locs[n])
found.append(n)
for n in found: ns.remove(n)
fr = fr.f_back
assert not ns, f'Failed to find {ns}'
if not hasattr(self, '__stored_args__'): self.__stored_args__ = {}
if attrs: return _store_attr(self, **attrs)

ns = re.split(', *', names) if names else args[1:]
_store_attr(self, **{n:fr.f_locals[n] for n in ns if n not in L(but)})

# Cell
def attrdict(o, *ks):
Expand Down Expand Up @@ -448,13 +444,13 @@ def log_args(f=None, *, to_return=False, but=None, but_as=None):
def _f(*args, **kwargs):
f_insp,args_insp = f,args
xtra_kwargs = {}
# some functions don't have correct signature (e.g. functions with @delegates such as Datasets.__init__) so we get the one from the class
# some functions don't have correct signature (e.g. functions with @delegates such as Datasets.__init__)
if '__init__' in f.__qualname__:
# from https://stackoverflow.com/a/25959545/3474490
cls = getattr(inspect.getmodule(f), f.__qualname__.split('.<locals>', 1)[0].rsplit('.', 1)[0]) # args[0].__class__ would not consider inheritance
# args[0].__class__ would not consider inheritance
cls = getattr(inspect.getmodule(f), f.__qualname__.split('.<locals>', 1)[0].rsplit('.', 1)[0])
f_insp, args_insp = cls, args[1:]
try:
func_args = inspect.signature(f_insp).bind(*args_insp, **kwargs)
try: func_args = inspect.signature(f_insp).bind(*args_insp, **kwargs)
except Exception as e:
try:
# sometimes it happens because the signature does not reference some kwargs
Expand Down Expand Up @@ -503,6 +499,7 @@ def __getattr__(self,k):
self.ready = False
return self

# Cell
class _SelfCls:
def __getattr__(self,k): return getattr(_Self(),k)
def __getitem__(self,i): return self.__getattr__('__getitem__')(i)
Expand Down

0 comments on commit ea1c33f

Please sign in to comment.