[Conda Site-Packages] Transformer changing the semantics of some programs

[rlnsanz]

This bug requires further investigation to identify the root cause.

On execution of the standard iris Scikit-Learn example, Flor produces the following error:

(base) ➜  iris_loop git:(master) flor python iris_raw.py complete_iris
Traceback (most recent call last):
  File "/anaconda3/envs/flor/lib/python3.7/site-packages/flor/commands/flython.py", line 38, in exec_flython
    spec.loader.exec_module(module)
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/Users/rogarcia/sandbox/iris_loop/iris_raw.py", line 60, in <module>
    from sklearn import datasets
  File "/anaconda3/envs/flor/lib/python3.7/site-packages/sklearn/datasets/__init__.py", line 23, in <module>
    from .twenty_newsgroups import fetch_20newsgroups
  File "/anaconda3/envs/flor/lib/python3.7/site-packages/sklearn/datasets/twenty_newsgroups.py", line 39, in <module>
    from ..feature_extraction.text import CountVectorizer
  File "/anaconda3/envs/flor/lib/python3.7/site-packages/sklearn/feature_extraction/__init__.py", line 9, in <module>
    from . import text
  File "/anaconda3/envs/flor/lib/python3.7/site-packages/sklearn/feature_extraction/text.py", line 18, in <module>
    from ..preprocessing import normalize
  File "/anaconda3/envs/flor/lib/python3.7/site-packages/sklearn/preprocessing/__init__.py", line 6, in <module>
    from .data import Binarizer
  File "/anaconda3/envs/flor/lib/python3.7/site-packages/sklearn/preprocessing/data.py", line 8, in <module>
    from scipy import stats
  File "/anaconda3/envs/flor/lib/python3.7/site-packages/scipy/stats/__init__.py", line 344, in <module>
    from .stats import *
  File "/anaconda3/envs/flor/lib/python3.7/site-packages/scipy/stats/stats.py", line 171, in <module>
    from . import distributions
  File "/anaconda3/envs/flor/lib/python3.7/site-packages/scipy/stats/distributions.py", line 3, in <module>
    from . import _continuous_distns
  File "/anaconda3/envs/flor/lib/python3.7/site-packages/scipy/stats/_continuous_distns.py", line 113, in <module>
    class norm_gen(rv_continuous):
  File "/anaconda3/envs/flor/lib/python3.7/site-packages/scipy/stats/_continuous_distns.py", line 175, in norm_gen
    `optimizer` argument is ignored.\n\n""")
  File "/anaconda3/envs/flor/lib/python3.7/site-packages/scipy/misc/doccer.py", line 159, in _doc
    start_of_notes = cls_docstring.find(notes_header)
AttributeError: 'NoneType' object has no attribute 'find'
'NoneType' object has no attribute 'find'

None of the files listed are to blame for the fault. This was confirmed by manually replacing the files mentioned in the ErrorMessage with the pre-transformed library files. The error still appeared.

The error message disappeared when we reverted scipy/stats/_distn_infrastructure.py with the pre-transformed version of the file. This means the re-written file is executing functions without generating an error locally, and changing the semantics of the program. Because the culprit file is so large, we have not been able to analyze it and identify the AST-Transformer bug.

On inspection of the log generated by the flor-execution of the iris example, we noticed that rv_continuous._updated_ctop_param and rv_generic._construct_argparser appeared frequently in the log records.

The current patch involves reverting scipy/stats/_distn_infrastructure.pyto the pre-transformed version. The error disappears, but the root cause has not been resolved.

[rlnsanz]

Related:

Traceback (most recent call last):
  File "/anaconda3/envs/flor/bin/pyflor", line 6, in <module>
    from flor.__main__ import main
  File "/anaconda3/envs/flor/lib/python3.7/site-packages/flor/__init__.py", line 11, in <module>
    from flor.__main__ import install
  File "/anaconda3/envs/flor/lib/python3.7/site-packages/flor/__main__.py", line 3, in <module>
    from flor.commands.flan import exec_flan
  File "/anaconda3/envs/flor/lib/python3.7/site-packages/flor/commands/flan.py", line 5, in <module>
    from flor.state_machine_generator import Visitor
  File "/anaconda3/envs/flor/lib/python3.7/site-packages/flor/state_machine_generator/__init__.py", line 1, in <module>
    from .visitor import Visitor
  File "/anaconda3/envs/flor/lib/python3.7/site-packages/flor/state_machine_generator/visitor.py", line 5, in <module>
    from flor.log_scanner.scanner import Scanner
  File "/anaconda3/envs/flor/lib/python3.7/site-packages/flor/log_scanner/scanner.py", line 2, in <module>
    import pandas as pd
  File "/anaconda3/envs/flor/lib/python3.7/site-packages/pandas/__init__.py", line 6, in <module>
    __import__(dependency)
  File "/anaconda3/envs/flor/lib/python3.7/site-packages/numpy/__init__.py", line 142, in <module>
    from . import core
  File "/anaconda3/envs/flor/lib/python3.7/site-packages/numpy/core/__init__.py", line 25, in <module>
    from . import multiarray
  File "/anaconda3/envs/flor/lib/python3.7/site-packages/numpy/core/multiarray.py", line 60, in <module>
    def empty_like(prototype, dtype=None, order=None, subok=None):
  File "/anaconda3/envs/flor/lib/python3.7/site-packages/numpy/core/overrides.py", line 380, in decorator
    verify, docs_from_dispatcher=docs_from_dispatcher)(implementation)
  File "/anaconda3/envs/flor/lib/python3.7/site-packages/numpy/core/overrides.py", line 258, in decorator
    add_docstring(implementation, dispatcher.__doc__)
TypeError: add_docstring() argument 2 must be str, not None

[rlnsanz]

Found one bug. The insertion of function logging prefixes precedes the docstring, so that invocations to __doc__ return None even when the docstring is defined.

[rlnsanz]

Fixed by moving docstring to line immediately following function definition.