Integration tests

[miguelgfierro]

Description

Integration and smoke tests

@bethz, @jainr In the code I would like to have the sequence:

1. create conda env
2. run smoke
3. run integration
4. remove conda

Beth told me that there might be a more elegant way of doing this. Can you please offer some guideline?

@saidbleik @sharatsc the scheduler is not working at the moment (you might have seen the emails to devops). As a temporal solution I thought of running this pipeline every time there is a PR to master. Feel free to propose another idea

Related Issues

#25

Checklist:

• My code follows the code style of this project, as detailed in our contribution guidelines.
• I have added tests.
• I have updated the documentation accordingly.

[bethz]

Running two jobs in one pipeline is an efficient and easy-to-maintain solution.
Here's an example of the approach I suggested:
https://dev.azure.com/best-practices/recommenders/_build/results?buildId=4777

I'll also look at what has been submitted.

[miguelgfierro]

Running two jobs in one pipeline is an efficient and easy-to-maintain solution.
Here's an example of the approach I suggested:
https://dev.azure.com/best-practices/recommenders/_build/results?buildId=4777

Is there a way to share the environment between jobs so we don't have to install conda twice?

[miguelgfierro]

@AbhiramE @caseyhong @catherine667 guys can you review? there are some minor changes that slightly affects what you are developing in #191 and #19

[jisooghd]

lgtm!

[miguelgfierro]

hey @AbhiramE there is an error test_gensen_local. It is related to #179

tests/integration/test_notebooks_sentence_similarity.py F                [100%]

=================================== FAILURES ===================================
______________________________ test_gensen_local _______________________________

notebooks = {'bert_encoder': '/data/home/nlpadmin/myagent/_work/6/s/scenarios/sentence_similarity/bert_encoder.ipynb', 'embedding_...y_embeddings_baseline': '/data/home/nlpadmin/myagent/_work/6/s/scenarios/sentence_similarity/baseline_deep_dive.ipynb'}

    @pytest.mark.gpu
    @pytest.mark.integration
    def test_gensen_local(notebooks):
        notebook_path = notebooks["gensen_local"]
        pm.execute_notebook(
            notebook_path,
            OUTPUT_NOTEBOOK,
            kernel_name=KERNEL_NAME,
            parameters=dict(
                max_epoch=1,
                config_filepath="../../scenarios/sentence_similarity/gensen_config.json",
>               base_data_path="../../data",
            ),
        )

tests/integration/test_notebooks_sentence_similarity.py:57: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/data/anaconda/envs/integration_gpu/lib/python3.6/site-packages/papermill/execute.py:101: in execute_notebook
    raise_for_execution_errors(nb, output_path)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

nb = {'cells': [{'cell_type': 'code', 'metadata': {'inputHidden': True, 'hide_input': True}, 'execution_count': None, 'sour...d_time': '2019-07-26T15:53:52.121638', 'duration': 386.199085, 'exception': True}}, 'nbformat': 4, 'nbformat_minor': 2}
output_path = 'output.ipynb'

    def raise_for_execution_errors(nb, output_path):
        """Assigned parameters into the appropriate place in the input notebook
    
        Parameters
        ----------
        nb : NotebookNode
           Executable notebook object
        output_path : str
           Path to write executed notebook
        """
        error = None
        for cell in nb.cells:
            if cell.get("outputs") is None:
                continue
    
            for output in cell.outputs:
                if output.output_type == "error":
                    error = PapermillExecutionError(
                        exec_count=cell.execution_count,
                        source=cell.source,
                        ename=output.ename,
                        evalue=output.evalue,
                        traceback=output.traceback,
                    )
                    break
    
        if error:
            # Write notebook back out with the Error Message at the top of the Notebook.
            error_msg = ERROR_MESSAGE_TEMPLATE % str(error.exec_count)
            error_msg_cell = nbformat.v4.new_code_cell(
                source="%%html\n" + error_msg,
                outputs=[
                    nbformat.v4.new_output(output_type="display_data", data={"text/html": error_msg})
                ],
                metadata={"inputHidden": True, "hide_input": True},
            )
            nb.cells = [error_msg_cell] + nb.cells
            write_ipynb(nb, output_path)
>           raise error
E           papermill.exceptions.PapermillExecutionError: 
E           ---------------------------------------------------------------------------
E           Exception encountered at "In [8]":
E           ---------------------------------------------------------------------------
E           FileNotFoundError                         Traceback (most recent call last)
E           /data/home/nlpadmin/myagent/_work/6/s/scenarios/sentence_similarity/gensen_wrapper.py in _validate_params(self)
E                56         try:
E           ---> 57             f = open(self.config_file)
E                58             self.config = self._read_config(self.config_file)
E           
E           FileNotFoundError: [Errno 2] No such file or directory: '../../scenarios/sentence_similarity/gensen_config.json'
E           
E           During handling of the above exception, another exception occurred:
E           
E           FileNotFoundError                         Traceback (most recent call last)
E           <ipython-input-8-542fca681a1f> in <module>
E                 3                        learning_rate = 0.0001,
E                 4                        cache_dir=base_data_path,
E           ----> 5                       max_epoch=max_epoch)
E           
E           /data/home/nlpadmin/myagent/_work/6/s/scenarios/sentence_similarity/gensen_wrapper.py in __init__(self, config_file, pretrained_embedding_path, learning_rate, cache_dir, max_epoch)
E                40         self.max_epoch = max_epoch
E                41 
E           ---> 42         self._validate_params()
E                43 
E                44     def _validate_params(self):
E           
E           /data/home/nlpadmin/myagent/_work/6/s/scenarios/sentence_similarity/gensen_wrapper.py in _validate_params(self)
E                59             f.close()
E                60         except FileNotFoundError:
E           ---> 61             raise FileNotFoundError("Provided config file does not exist!")
E                62 
E                63     def _get_gensen_tokens(self, train_df=None, dev_df=None, test_df=None):
E           
E           FileNotFoundError: Provided config file does not exist!

do you know what the problem could be?

[catherine667]

I will wait until you have merged this PR and then change my GenSen #199 and BERT #191 notebook testing accordingly. Now, my tests broke if I change it to integration tests.

Others, LGTM!

[AbhiramE]

hey @AbhiramE there is an error test_gensen_local. It is related to #179

tests/integration/test_notebooks_sentence_similarity.py F                [100%]

=================================== FAILURES ===================================
______________________________ test_gensen_local _______________________________

notebooks = {'bert_encoder': '/data/home/nlpadmin/myagent/_work/6/s/scenarios/sentence_similarity/bert_encoder.ipynb', 'embedding_...y_embeddings_baseline': '/data/home/nlpadmin/myagent/_work/6/s/scenarios/sentence_similarity/baseline_deep_dive.ipynb'}

    @pytest.mark.gpu
    @pytest.mark.integration
    def test_gensen_local(notebooks):
        notebook_path = notebooks["gensen_local"]
        pm.execute_notebook(
            notebook_path,
            OUTPUT_NOTEBOOK,
            kernel_name=KERNEL_NAME,
            parameters=dict(
                max_epoch=1,
                config_filepath="../../scenarios/sentence_similarity/gensen_config.json",
>               base_data_path="../../data",
            ),
        )

tests/integration/test_notebooks_sentence_similarity.py:57: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/data/anaconda/envs/integration_gpu/lib/python3.6/site-packages/papermill/execute.py:101: in execute_notebook
    raise_for_execution_errors(nb, output_path)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

nb = {'cells': [{'cell_type': 'code', 'metadata': {'inputHidden': True, 'hide_input': True}, 'execution_count': None, 'sour...d_time': '2019-07-26T15:53:52.121638', 'duration': 386.199085, 'exception': True}}, 'nbformat': 4, 'nbformat_minor': 2}
output_path = 'output.ipynb'

    def raise_for_execution_errors(nb, output_path):
        """Assigned parameters into the appropriate place in the input notebook
    
        Parameters
        ----------
        nb : NotebookNode
           Executable notebook object
        output_path : str
           Path to write executed notebook
        """
        error = None
        for cell in nb.cells:
            if cell.get("outputs") is None:
                continue
    
            for output in cell.outputs:
                if output.output_type == "error":
                    error = PapermillExecutionError(
                        exec_count=cell.execution_count,
                        source=cell.source,
                        ename=output.ename,
                        evalue=output.evalue,
                        traceback=output.traceback,
                    )
                    break
    
        if error:
            # Write notebook back out with the Error Message at the top of the Notebook.
            error_msg = ERROR_MESSAGE_TEMPLATE % str(error.exec_count)
            error_msg_cell = nbformat.v4.new_code_cell(
                source="%%html\n" + error_msg,
                outputs=[
                    nbformat.v4.new_output(output_type="display_data", data={"text/html": error_msg})
                ],
                metadata={"inputHidden": True, "hide_input": True},
            )
            nb.cells = [error_msg_cell] + nb.cells
            write_ipynb(nb, output_path)
>           raise error
E           papermill.exceptions.PapermillExecutionError: 
E           ---------------------------------------------------------------------------
E           Exception encountered at "In [8]":
E           ---------------------------------------------------------------------------
E           FileNotFoundError                         Traceback (most recent call last)
E           /data/home/nlpadmin/myagent/_work/6/s/scenarios/sentence_similarity/gensen_wrapper.py in _validate_params(self)
E                56         try:
E           ---> 57             f = open(self.config_file)
E                58             self.config = self._read_config(self.config_file)
E           
E           FileNotFoundError: [Errno 2] No such file or directory: '../../scenarios/sentence_similarity/gensen_config.json'
E           
E           During handling of the above exception, another exception occurred:
E           
E           FileNotFoundError                         Traceback (most recent call last)
E           <ipython-input-8-542fca681a1f> in <module>
E                 3                        learning_rate = 0.0001,
E                 4                        cache_dir=base_data_path,
E           ----> 5                       max_epoch=max_epoch)
E           
E           /data/home/nlpadmin/myagent/_work/6/s/scenarios/sentence_similarity/gensen_wrapper.py in __init__(self, config_file, pretrained_embedding_path, learning_rate, cache_dir, max_epoch)
E                40         self.max_epoch = max_epoch
E                41 
E           ---> 42         self._validate_params()
E                43 
E                44     def _validate_params(self):
E           
E           /data/home/nlpadmin/myagent/_work/6/s/scenarios/sentence_similarity/gensen_wrapper.py in _validate_params(self)
E                59             f.close()
E                60         except FileNotFoundError:
E           ---> 61             raise FileNotFoundError("Provided config file does not exist!")
E                62 
E                63     def _get_gensen_tokens(self, train_df=None, dev_df=None, test_df=None):
E           
E           FileNotFoundError: Provided config file does not exist!

do you know what the problem could be?

The problem could be the relative path I am providing to the file. I was providing paths relative to the integration folder. Debugging this now. Should be able to fix it soon.

[AbhiramE]

Made minor changes to path in gensen local test. Approved if tests run fine. :)

[AbhiramE]

Seeing this error in the logs. Is there a problem with the test setup?

Traceback (most recent call last):
  File "/data/anaconda/bin/pytest", line 7, in <module>
    from py.test import main
ModuleNotFoundError: No module named 'py'
##[error]Bash exited with code '1'.
##[section]Finishing: Run Unit tests

[miguelgfierro]

Seeing this error in the logs. Is there a problem with the test setup?

For some reason the nlp_gpu env of the test machine was removed. Kicking the tests again