Table.__getattr__ issue when calling np.array on a Table row

[chrispyles]

In grading CP 298 project 3, we have a table like this:

power       | captain    | weve        | move        | three       | world       | ship       | system      | men         | head        | marri       | huh         | hous        | nice        | home        | boy         | miss        | play        | mother      | hello
0.000540833 | 0          | 0           | 0.00108167  | 0           | 0.000540833 | 0          | 0.000540833 | 0           | 0           | 0           | 0.000540833 | 0           | 0           | 0.00108167  | 0           | 0           | 0           | 0.0016225   | 0
0.000705219 | 0          | 0.000705219 | 0.00105783  | 0           | 0.000705219 | 0          | 0           | 0           | 0.00105783  | 0           | 0.00246826  | 0.000352609 | 0.00105783  | 0.000352609 | 0.00141044  | 0.00211566  | 0           | 0           | 0
0.000237248 | 0.00118624 | 0.000711744 | 0.000237248 | 0.000474496 | 0.000948992 | 0.00260973 | 0           | 0.00166073  | 0.000711744 | 0.000711744 | 0           | 0           | 0.000948992 | 0           | 0           | 0.00284697  | 0           | 0           | 0.000474496
0.000329815 | 0          | 0.000329815 | 0.000329815 | 0.000659631 | 0.000329815 | 0          | 0           | 0           | 0           | 0           | 0.000329815 | 0.000329815 | 0.000329815 | 0.000659631 | 0.000989446 | 0.00131926  | 0.000329815 | 0           | 0
0.0041645   | 0          | 0           | 0.00104112  | 0.000260281 | 0.000780843 | 0          | 0           | 0.000260281 | 0.000780843 | 0.000260281 | 0           | 0.00130141  | 0.00156169  | 0.00104112  | 0           | 0.000780843 | 0           | 0.000260281 | 0.000260281
0.000836587 | 0          | 0.000836587 | 0           | 0.000557724 | 0.000836587 | 0          | 0.000836587 | 0           | 0.00111545  | 0.000557724 | 0           | 0           | 0           | 0.000278862 | 0.000557724 | 0.000278862 | 0.00278862  | 0.00585611  | 0.000836587
0.00038994  | 0          | 0.000779879 | 0.00175473  | 0.000584909 | 0.00136479  | 0.00019497 | 0           | 0.00136479  | 0.000779879 | 0           | 0.00019497  | 0.00019497  | 0           | 0           | 0.00038994  | 0.000584909 | 0.000779879 | 0.00019497  | 0.00038994
0           | 0.00386598 | 0.00257732  | 0.000859107 | 0           | 0.00128866  | 0.00300687 | 0.00257732  | 0           | 0.00171821  | 0.000429553 | 0           | 0           | 0.000429553 | 0           | 0.00558419  | 0           | 0.000859107 | 0.000429553 | 0
0.00348953  | 0          | 0.000498504 | 0           | 0           | 0.00149551  | 0          | 0           | 0.000997009 | 0.000498504 | 0           | 0.000498504 | 0           | 0           | 0.000997009 | 0           | 0.000498504 | 0           | 0.000498504 | 0
0           | 0.00042526 | 0.00127578  | 0.00021263  | 0.00042526  | 0.00021263  | 0          | 0           | 0.00042526  | 0           | 0.00042526  | 0           | 0.00127578  | 0.00276419  | 0.0021263   | 0.00042526  | 0.00042526  | 0.00042526  | 0.000637891 | 0.00042526

and when the following Table.apply is run on the above table (train_f)

train_f.apply(lambda r: np.sum(np.abs(np.array(r))) == 0)

the following error is thrown:

Exception raised:
    Traceback (most recent call last):
      File "/usr/lib/python3.6/doctest.py", line 1330, in __run
        compileflags, 1), test.globs)
      File "<doctest /autograder/source/tests/q3_1_1.py 2[1]>", line 1, in <module>
        train_f.apply(lambda r: np.sum(np.abs(np.array(r))) == 0)
      File "/usr/local/lib/python3.6/dist-packages/datascience/tables.py", line 359, in apply
        return np.array([fn(row) for row in self.rows])
      File "/usr/local/lib/python3.6/dist-packages/datascience/tables.py", line 359, in <listcomp>
        return np.array([fn(row) for row in self.rows])
      File "<doctest /autograder/source/tests/q3_1_1.py 2[1]>", line 1, in <lambda>
        train_f.apply(lambda r: np.sum(np.abs(np.array(r))) == 0)
      File "/usr/local/lib/python3.6/dist-packages/datascience/tables.py", line 2885, in __getattr__
        return self[self._table.column_index(column_label)]
      File "/usr/local/lib/python3.6/dist-packages/datascience/tables.py", line 306, in column_index
        return self.labels.index(label)
    ValueError: tuple.index(x): x not in tuple

I've narrowed down the issue to something with numpy==1.18.4 but I'm not sure what. When I downgrade numpy to 1.16.0 in the autograding container, the test passes. Would be worthwhile to look into this bug so that datascience is compatible with numpy>1.16.0.

[chrispyles]

Upon further investigation, this is the cause of the issue noted on the 8X Hub by @manikui. I've opened a PR to datahub to downgrade numpy to 1.16.0 on the 8X Hub.

[davidwagner]

Thanks. It sounds like the problem is no longer present on 8X. However, I'm concerned about fixing this by downgrading numpy. If the datascience package has issues with the latest version of numpy, I'd like to understand that, as we can't stay on old versions of numpy forever -- we should try to make sure the datascience package works well with the latest versions of numpy. If someone has time to try to reproduce this with the latest version of numpy and minimize it to figure out if we're going to have a problem with newer versions of numpy, that might be useful.

[adnanhemani]

So I've narrowed the scope of the problem: The behavior of np.array(...) has changed from NumPy v1.17.5 to v1.18.0: up to v1.17.5, if we tried to convert a Row object (which at the backend is a named tuple object) to a numpy array via np.array, it'd work fine. However, when doing the same operation in NumPy v1.18.0, the operation errors.

I haven't been able to pinpoint why this is the case, how to fix this, or what code changes on the NumPy side have caused this change in behavior but I will continue working on it.

[adnanhemani]

Hi folks, I've found and made the changes. Please take a look and review the PR (#427 ). Thanks!