Expand Setup Tests

[dmitrypolo]

Part of our Data Days for Good initiative was to work on this repo. One of the requested additions was fleshing out the setup tests a little bit more to test for what would happen if user input was introduced. This was mentioned by Isaac. I have separated out the fixture and put it in it's own file, which is discoverable by pytest and mentioned in the docs. I have also increased the scope of creating the temp directory to be a class scope so it doesn't have to create it and tear it down after every function call. Lastly I also test for certain conditions given user input.

[dmitrypolo]

@isms @pjbull please review as I see you are the two most active users in this repo

[isms]

@dmitrypolo Will review today, thanks!

[isms]

@dmitrypolo When I run pyt.test with a fresh pull and fresh virtual environment (Python 3.6) I get the following:

$ py.test
====================================== test session starts ======================================
platform linux -- Python 3.6.3, pytest-3.6.3, py-1.5.4, pluggy-0.6.0
rootdir: /home/isaac/Documents/cookiecutter-data-science, inifile:
collected 18 items                                                                              

tests/test_creation.py .........FFFFFFFFF                                                 [100%]

=========================================== FAILURES ============================================
___________________ TestCookieSetup.test_project_name[default_baked_project1] ___________________

self = <test_creation.TestCookieSetup object at 0x7f5562677828>

    def test_project_name(self):
        project = self.path
        if pytest.param.get('project_name'):
>           assert project.name == 'DataDriven'
E           AssertionError: assert 'DrivenData' == 'DataDriven'
E             - DrivenData
E             + DataDriven

tests/test_creation.py:29: AssertionError
______________________ TestCookieSetup.test_author[default_baked_project1] ______________________

self = <test_creation.TestCookieSetup object at 0x7f55626773c8>

    def test_author(self):
        setup_ = self.path / 'setup.py'
        args = ['python', setup_, '--author']
>       p = check_output(args).decode('ascii').strip()

tests/test_creation.py:36: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/lib/python3.6/subprocess.py:336: in check_output
    **kwargs).stdout
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

input = None, timeout = None, check = True
popenargs = (['python', PosixPath('/tmp/pytest-of-isaac/pytest-0/data-project0/DrivenData/setup.py'), '--author'],)
kwargs = {'stdout': -1}, process = <subprocess.Popen object at 0x7f5562677dd8>, stdout = b''
stderr = None, retcode = 2

    def run(*popenargs, input=None, timeout=None, check=False, **kwargs):
        """Run command with arguments and return a CompletedProcess instance.
    
        The returned instance will have attributes args, returncode, stdout and
        stderr. By default, stdout and stderr are not captured, and those attributes
        will be None. Pass stdout=PIPE and/or stderr=PIPE in order to capture them.
    
        If check is True and the exit code was non-zero, it raises a
        CalledProcessError. The CalledProcessError object will have the return code
        in the returncode attribute, and output & stderr attributes if those streams
        were captured.
    
        If timeout is given, and the process takes too long, a TimeoutExpired
        exception will be raised.
    
        There is an optional argument "input", allowing you to
        pass a string to the subprocess's stdin.  If you use this argument
        you may not also use the Popen constructor's "stdin" argument, as
        it will be used internally.
    
        The other arguments are the same as for the Popen constructor.
    
        If universal_newlines=True is passed, the "input" argument must be a
        string and stdout/stderr in the returned object will be strings rather than
        bytes.
        """
        if input is not None:
            if 'stdin' in kwargs:
                raise ValueError('stdin and input arguments may not both be used.')
            kwargs['stdin'] = PIPE
    
        with Popen(*popenargs, **kwargs) as process:
            try:
                stdout, stderr = process.communicate(input, timeout=timeout)
            except TimeoutExpired:
                process.kill()
                stdout, stderr = process.communicate()
                raise TimeoutExpired(process.args, timeout, output=stdout,
                                     stderr=stderr)
            except:
                process.kill()
                process.wait()
                raise
            retcode = process.poll()
            if check and retcode:
                raise CalledProcessError(retcode, process.args,
>                                        output=stdout, stderr=stderr)
E               subprocess.CalledProcessError: Command '['python', PosixPath('/tmp/pytest-of-isaac/pytest-0/data-project0/DrivenData/setup.py'), '--author']' returned non-zero exit status 2.

/usr/lib/python3.6/subprocess.py:418: CalledProcessError
------------------------------------- Captured stderr call --------------------------------------
python: can't open file '/tmp/pytest-of-isaac/pytest-0/data-project0/DrivenData/setup.py': [Errno 2] No such file or directory
______________________ TestCookieSetup.test_readme[default_baked_project1] ______________________

self = <test_creation.TestCookieSetup object at 0x7f5564650390>

    def test_readme(self):
        readme_path = self.path / 'README.md'
>       assert readme_path.exists()
E       AssertionError: assert False
E        +  where False = <bound method Path.exists of PosixPath('/tmp/pytest-of-isaac/pytest-0/data-project0/DrivenData/README.md')>()
E        +    where <bound method Path.exists of PosixPath('/tmp/pytest-of-isaac/pytest-0/data-project0/DrivenData/README.md')> = PosixPath('/tmp/pytest-of-isaac/pytest-0/data-project0/DrivenData/README.md').exists

tests/test_creation.py:44: AssertionError
______________________ TestCookieSetup.test_setup[default_baked_project1] _______________________

self = <test_creation.TestCookieSetup object at 0x7f55626614a8>

    def test_setup(self):
        setup_ = self.path / 'setup.py'
        args = ['python', setup_, '--version']
>       p = check_output(args).decode('ascii').strip()

tests/test_creation.py:53: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/lib/python3.6/subprocess.py:336: in check_output
    **kwargs).stdout
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

input = None, timeout = None, check = True
popenargs = (['python', PosixPath('/tmp/pytest-of-isaac/pytest-0/data-project0/DrivenData/setup.py'), '--version'],)
kwargs = {'stdout': -1}, process = <subprocess.Popen object at 0x7f5562661390>, stdout = b''
stderr = None, retcode = 2

    def run(*popenargs, input=None, timeout=None, check=False, **kwargs):
        """Run command with arguments and return a CompletedProcess instance.
    
        The returned instance will have attributes args, returncode, stdout and
        stderr. By default, stdout and stderr are not captured, and those attributes
        will be None. Pass stdout=PIPE and/or stderr=PIPE in order to capture them.
    
        If check is True and the exit code was non-zero, it raises a
        CalledProcessError. The CalledProcessError object will have the return code
        in the returncode attribute, and output & stderr attributes if those streams
        were captured.
    
        If timeout is given, and the process takes too long, a TimeoutExpired
        exception will be raised.
    
        There is an optional argument "input", allowing you to
        pass a string to the subprocess's stdin.  If you use this argument
        you may not also use the Popen constructor's "stdin" argument, as
        it will be used internally.
    
        The other arguments are the same as for the Popen constructor.
    
        If universal_newlines=True is passed, the "input" argument must be a
        string and stdout/stderr in the returned object will be strings rather than
        bytes.
        """
        if input is not None:
            if 'stdin' in kwargs:
                raise ValueError('stdin and input arguments may not both be used.')
            kwargs['stdin'] = PIPE
    
        with Popen(*popenargs, **kwargs) as process:
            try:
                stdout, stderr = process.communicate(input, timeout=timeout)
            except TimeoutExpired:
                process.kill()
                stdout, stderr = process.communicate()
                raise TimeoutExpired(process.args, timeout, output=stdout,
                                     stderr=stderr)
            except:
                process.kill()
                process.wait()
                raise
            retcode = process.poll()
            if check and retcode:
                raise CalledProcessError(retcode, process.args,
>                                        output=stdout, stderr=stderr)
E               subprocess.CalledProcessError: Command '['python', PosixPath('/tmp/pytest-of-isaac/pytest-0/data-project0/DrivenData/setup.py'), '--version']' returned non-zero exit status 2.

/usr/lib/python3.6/subprocess.py:418: CalledProcessError
------------------------------------- Captured stderr call --------------------------------------
python: can't open file '/tmp/pytest-of-isaac/pytest-0/data-project0/DrivenData/setup.py': [Errno 2] No such file or directory
_____________________ TestCookieSetup.test_license[default_baked_project1] ______________________

self = <test_creation.TestCookieSetup object at 0x7f55625a4278>

    def test_license(self):
        license_path = self.path / 'LICENSE'
>       assert license_path.exists()
E       AssertionError: assert False
E        +  where False = <bound method Path.exists of PosixPath('/tmp/pytest-of-isaac/pytest-0/data-project0/DrivenData/LICENSE')>()
E        +    where <bound method Path.exists of PosixPath('/tmp/pytest-of-isaac/pytest-0/data-project0/DrivenData/LICENSE')> = PosixPath('/tmp/pytest-of-isaac/pytest-0/data-project0/DrivenData/LICENSE').exists

tests/test_creation.py:58: AssertionError
___________________ TestCookieSetup.test_license_type[default_baked_project1] ___________________

self = <test_creation.TestCookieSetup object at 0x7f55625b9208>

    def test_license_type(self):
        setup_ = self.path / 'setup.py'
        args = ['python', setup_, '--license']
>       p = check_output(args).decode('ascii').strip()

tests/test_creation.py:64: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/lib/python3.6/subprocess.py:336: in check_output
    **kwargs).stdout
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

input = None, timeout = None, check = True
popenargs = (['python', PosixPath('/tmp/pytest-of-isaac/pytest-0/data-project0/DrivenData/setup.py'), '--license'],)
kwargs = {'stdout': -1}, process = <subprocess.Popen object at 0x7f55625b91d0>, stdout = b''
stderr = None, retcode = 2

    def run(*popenargs, input=None, timeout=None, check=False, **kwargs):
        """Run command with arguments and return a CompletedProcess instance.
    
        The returned instance will have attributes args, returncode, stdout and
        stderr. By default, stdout and stderr are not captured, and those attributes
        will be None. Pass stdout=PIPE and/or stderr=PIPE in order to capture them.
    
        If check is True and the exit code was non-zero, it raises a
        CalledProcessError. The CalledProcessError object will have the return code
        in the returncode attribute, and output & stderr attributes if those streams
        were captured.
    
        If timeout is given, and the process takes too long, a TimeoutExpired
        exception will be raised.
    
        There is an optional argument "input", allowing you to
        pass a string to the subprocess's stdin.  If you use this argument
        you may not also use the Popen constructor's "stdin" argument, as
        it will be used internally.
    
        The other arguments are the same as for the Popen constructor.
    
        If universal_newlines=True is passed, the "input" argument must be a
        string and stdout/stderr in the returned object will be strings rather than
        bytes.
        """
        if input is not None:
            if 'stdin' in kwargs:
                raise ValueError('stdin and input arguments may not both be used.')
            kwargs['stdin'] = PIPE
    
        with Popen(*popenargs, **kwargs) as process:
            try:
                stdout, stderr = process.communicate(input, timeout=timeout)
            except TimeoutExpired:
                process.kill()
                stdout, stderr = process.communicate()
                raise TimeoutExpired(process.args, timeout, output=stdout,
                                     stderr=stderr)
            except:
                process.kill()
                process.wait()
                raise
            retcode = process.poll()
            if check and retcode:
                raise CalledProcessError(retcode, process.args,
>                                        output=stdout, stderr=stderr)
E               subprocess.CalledProcessError: Command '['python', PosixPath('/tmp/pytest-of-isaac/pytest-0/data-project0/DrivenData/setup.py'), '--license']' returned non-zero exit status 2.

/usr/lib/python3.6/subprocess.py:418: CalledProcessError
------------------------------------- Captured stderr call --------------------------------------
python: can't open file '/tmp/pytest-of-isaac/pytest-0/data-project0/DrivenData/setup.py': [Errno 2] No such file or directory
___________________ TestCookieSetup.test_requirements[default_baked_project1] ___________________

self = <test_creation.TestCookieSetup object at 0x7f5562640390>

    def test_requirements(self):
        reqs_path = self.path / 'requirements.txt'
>       assert reqs_path.exists()
E       AssertionError: assert False
E        +  where False = <bound method Path.exists of PosixPath('/tmp/pytest-of-isaac/pytest-0/data-project0/DrivenData/requirements.txt')>()
E        +    where <bound method Path.exists of PosixPath('/tmp/pytest-of-isaac/pytest-0/data-project0/DrivenData/requirements.txt')> = PosixPath('/tmp/pytest-of-isaac/pytest-0/data-project0/DrivenData/requirements.txt').exists

tests/test_creation.py:72: AssertionError
_____________________ TestCookieSetup.test_makefile[default_baked_project1] _____________________

self = <test_creation.TestCookieSetup object at 0x7f5562677f98>

    def test_makefile(self):
        makefile_path = self.path / 'Makefile'
>       assert makefile_path.exists()
E       AssertionError: assert False
E        +  where False = <bound method Path.exists of PosixPath('/tmp/pytest-of-isaac/pytest-0/data-project0/DrivenData/Makefile')>()
E        +    where <bound method Path.exists of PosixPath('/tmp/pytest-of-isaac/pytest-0/data-project0/DrivenData/Makefile')> = PosixPath('/tmp/pytest-of-isaac/pytest-0/data-project0/DrivenData/Makefile').exists

tests/test_creation.py:81: AssertionError
_____________________ TestCookieSetup.test_folders[default_baked_project1] ______________________

self = <test_creation.TestCookieSetup object at 0x7f5562642390>

    def test_folders(self):
        expected_dirs = [
            'data',
            'data/external',
            'data/interim',
            'data/processed',
            'data/raw',
            'docs',
            'models',
            'notebooks',
            'references',
            'reports',
            'reports/figures',
            'src',
            'src/data',
            'src/features',
            'src/models',
            'src/visualization',
        ]
    
        ignored_dirs = [
            str(self.path)
        ]
    
        abs_expected_dirs = [str(self.path / d) for d in expected_dirs]
>       abs_dirs, _, _ = list(zip(*os.walk(self.path)))
E       ValueError: not enough values to unpack (expected 3, got 0)

tests/test_creation.py:109: ValueError
============================== 9 failed, 9 passed in 1.47 seconds ===============================

Any insight?

[dmitrypolo]

I had the name backwards, for the assertion, can you please try again? If it fails on the temp file creation then I might have to look into it more as it might be a difference between Darwin and Linux

[dmitrypolo]

seems like it is an issue between Linux and Mac, just hopped onto a Linux box and verified the same errors are happening. I will investigate and submit a fix when ready. Thanks for pointing this out. @isms

[dmitrypolo]

@isms I figured out the issue and tested on Linux and all tests came back good.

platform linux -- Python 3.6.1, pytest-3.6.3, py-1.5.4, pluggy-0.6.0
rootdir: /cookies/cookiecutter-data-science, inifile:
collected 18 items                                                                                                               

tests/test_creation.py ..................                                                                                  [100%]

=================================================== 18 passed in 1.53 seconds ====================================================

The issue is with how the project is created on Mac/Darwin vs. Linux. On Mac when you supply a dictionary of arguments and provide it a project_name, i.e DrivenData the tmpdir_factory will make a file something to the extent of /private/tmp/something/DrivenData. However, on Linux this same function will yield /tmp/something/drivendata. It was making the basename lower case so the subprocess could not find the file because it was searching in the wrong place. There is a check in place now that identifies the system and processes accordingly. Please let me know your thoughts, feedbacks, and requested improvements. Thanks!

[isms]

Merged, thanks @dmitrypolo!