Add unit tests

[SebastianZimmeck]

We would like to add some testing code. @kalicki1 will add a few unit tests to start. Especially, let's test PFP's analysis functionality in analysis.py and privacy_practices.py.

A good starting point are Python's pytest and unittest frameworks. Here is a tutorial that covers both.

For the time being, we can just focus on passing tests and not worry about failing tests. As @rgoldstein01 mentioned, for example, if you want to test whether a function recognizes certain keywords in files, you can create some dummy input that represents an app file (e.g., a text file with certain keywords).

You can create a new directory tests under privacyflash-pro and include your files there.

(cc'ing @davebaraka, @pakaelbling)

[SebastianZimmeck]

@kalicki1, I created a new branch issue-5 for your work.

[SebastianZimmeck]

I added a placeholder in the readme for the instructions how to perform the testing. @kalicki1, could you add the instructions once you have finalized your unit tests?

[kalicki1]

I just updated the instructions for running the current tests, @SebastianZimmeck . I will update them as more tests get added and if the process changes over time.

[SebastianZimmeck]

Thanks, @kalicki1. I followed your instructions and got the following error:

szimmeck@Starship ~/D/C/privacyflash-pro> python3 -m unittest
E
======================================================================
ERROR: test_privacy_practices (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: test_privacy_practices
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/unittest/loader.py", line 436, in _find_test_path
    module = self._get_module_from_name(name)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/unittest/loader.py", line 377, in _get_module_from_name
    __import__(name)
  File "/Users/szimmeck/Desktop/Code/privacyflash-pro/test_privacy_practices.py", line 13, in <module>
    from policygenerator.privacy_practices import retrieve_privacy_practice_data
  File "/Users/szimmeck/Desktop/Code/privacyflash-pro/policygenerator/privacy_practices.py", line 17, in <module>
    from constants import Practices
ModuleNotFoundError: No module named 'constants'


----------------------------------------------------------------------
Ran 1 test in 0.000s

FAILED (errors=1) 

Perhaps, this error is related to moving the files per your instructions. When I am using from policygenerator.constants import Practices instead of just from constants import Practices in privacy_practices.py, the tests are running.

Can you look into how to run the tests without moving files around or changing the files of PrivacyFlash Pro (e.g., is this relevant?)

[SebastianZimmeck]

If anyone else has a solution, please chime in as well.

(cc'ing @rgoldstein01)

[SebastianZimmeck]

@davebaraka and @rgoldstein01, I looked a little bit into this. It seems that we cannot import modules using an absolute path. For example, in privacy_practices.py, when I am changing line 17 from

constants import Practices

to

policygenerator.constants import Practices,

I am not seeing any error. However, when I am running PFP with this change on the AdColony test project, the results are not displayed. Do you have any thoughts? Is there code in the analysis or the UI that needs to be changed as well?

[SebastianZimmeck]

@kalicki1, as we just discussed, when I am running the tests, five of them are failing. At the moment, I am thinking that we should have passing tests. It seems that you are getting different results, though. I am having some errors. Here is my output:

szimmeck@Starship ~/D/C/privacyflash-pro> python3 -m unittest
FFF/Users/szimmeck/Library/Python/3.7/lib/python/site-packages/PyYAML-3.12-py3.7-macosx-10.9-x86_64.egg/yaml/constructor.py:126: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
F...F
======================================================================
FAIL: test_get_cart_loc (tests.test_privacy_practices.TestPrivacyPractices)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/szimmeck/Desktop/Code/privacyflash-pro/tests/test_privacy_practices.py", line 82, in test_get_cart_loc
    self.assertEqual(len(res_dir), no_dir) # check length against predetermined count
AssertionError: 0 != 5

======================================================================
FAIL: test_get_pod_loc (tests.test_privacy_practices.TestPrivacyPractices)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/szimmeck/Desktop/Code/privacyflash-pro/tests/test_privacy_practices.py", line 65, in test_get_pod_loc
    self.assertEqual(len(res_dir), no_dir) # check length against predetermined count
AssertionError: 0 != 2

======================================================================
FAIL: test_grab_third_party_files (tests.test_privacy_practices.TestPrivacyPractices)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/szimmeck/Desktop/Code/privacyflash-pro/tests/test_privacy_practices.py", line 100, in test_grab_third_party_files
    self.assertEqual(len(res_dir[path]), no_dir) # check len against predetermined count
AssertionError: 5 != 6

======================================================================
FAIL: test_load_data (tests.test_privacy_practices.TestPrivacyPractices)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/szimmeck/Desktop/Code/privacyflash-pro/tests/test_privacy_practices.py", line 150, in test_load_data
    self.assertEqual(len(fp_files), 25) 			# test_search_root_dir
AssertionError: 20 != 25

======================================================================
FAIL: test_search_root_dir (tests.test_privacy_practices.TestPrivacyPractices)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/szimmeck/Desktop/Code/privacyflash-pro/tests/test_privacy_practices.py", line 48, in test_search_root_dir
    self.assertEqual(len(res_dir), no_files) # check length against predetermined count
AssertionError: 20 != 25

----------------------------------------------------------------------
Ran 8 tests in 0.476s

FAILED (failures=5) 

[davebaraka]

It seems that we cannot import modules using an absolute path.

I could not get eel to work when trying to structure the files as a package. The only way I could get eel to work is if it was left as a script. So as a potential solution, I put all the analysis files in a src folder and structured it as a python package. This allows us to include imports as expected.

I think everything should still be working.

[SebastianZimmeck]

Thanks, @davebaraka! I just added an init.py file to the src directory because running the tests was not working for me without it. The absolute/relative import point seems to be fixed now.

@kalicki1, the problem with the failing tests is still open, though. The output I am getting does not look quite right to me.

[SebastianZimmeck]

News here, @kalicki1?

[kalicki1]

@SebastianZimmeck, I fixed the issue with the last three test errors you had. My tests accounted for the fact that I had extra .DS_Store files on my machine that others may not have, so I made sure the tests did not include them.

However I still have not figured out what returned the first two errors, since I would expect the same error to come up under the FAIL: test_load_data flag, as it runs almost the same code. I am referring to self.assertEqual(len(sdk_files), 2 + 5) on line 170 in test_privacy_practices.py. I will keep looking, let me know if you are still getting the same error.

[SebastianZimmeck]

Thanks, @kalicki1. This is what I am currently getting:

szimmeck@Starship ~/D/C/privacyflash-pro> python3 -m unittest
FF./Users/szimmeck/Library/Python/3.7/lib/python/site-packages/PyYAML-3.12-py3.7-macosx-10.9-x86_64.egg/yaml/constructor.py:126: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
F....
======================================================================
FAIL: test_get_cart_loc (tests.test_privacy_practices.TestPrivacyPractices)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/szimmeck/Desktop/Code/privacyflash-pro/tests/test_privacy_practices.py", line 93, in test_get_cart_loc
    self.assertEqual(len(res_dir), no_dir) # check length against predetermined count
AssertionError: 0 != 5

======================================================================
FAIL: test_get_pod_loc (tests.test_privacy_practices.TestPrivacyPractices)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/szimmeck/Desktop/Code/privacyflash-pro/tests/test_privacy_practices.py", line 76, in test_get_pod_loc
    self.assertEqual(len(res_dir), no_dir) # check length against predetermined count
AssertionError: 0 != 2

======================================================================
FAIL: test_load_data (tests.test_privacy_practices.TestPrivacyPractices)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/szimmeck/Desktop/Code/privacyflash-pro/tests/test_privacy_practices.py", line 170, in test_load_data
    self.assertEqual(len(sdk_files), 2 + 5)			# test_grab_third_party_files
AssertionError: 0 != 7

----------------------------------------------------------------------
Ran 8 tests in 0.482s

FAILED (failures=3)

[SebastianZimmeck]

@kalicki1, can you try fixing the remaining errors by Thursday? If it is not possible, you can just remove the test cases that are throwing errors. Then, open a pull request please.