Testing Suite Updates

[Ethan-DeBandi99]

The ideas conveyed here were previously explored, but updates were not completed. Arkouda needs more robust scalable testing that interfaces with our benchmarking for simplified maintenance. Key components are listed below:

• Single configuration between testing and benchmarks.
• Configurable tests. This is already supported in pytest-benchmark, but needs to be expanded to pytest to make sure we have more code coverage and better testing on different scales.
• JSON output of test results for more flexible reporting and review.

Steps to Complete
Issues will be added for these at a later date.

• Remove any of the prototype code from the previous research into this so that we have a clean slate to start. (Remove Old Testing Prototype Code #2500)

• Configure the architecture and configuration files. This will include pytest.ini and conftest.py. We will only want to maintain one set of configuration for testing and benchmarking. Benchmarking will be turned off by default and will be manually enabled when needed. (New Testing Architecture #2504)

• Configure tests to scale and use the same/similar parameters for problem size and configuration as the benchmarks (Configuration included as part of New Testing Architecture #2504. This will require ongoing updates to handle different cases).

• Ensure the Benchmark Correctness checks from old benchmarking system are configured and runnable.

• Convert all tests to new format
  Simple Conversions

  • alignment_test.py Conversion for new test framework #2536
  • array_view_test.py conversion for new test framework #2537
  • bigint_agg_test.py Conversion for new test framework #2570
  • bitops_test.py Conversion for new test framework #2572
  • client_dtypes_test.py conversion for new test framework #2583
  • dtypes_test.py Conversion for new test framework #2605
  • index_test.py Conversion for new test framework #2616
  • indexing_test.py Conversion for new test framework #2620
  • join_tests.py Conversion for new test framework #2626
  • logger_test.py Conversion for new test framework #2648
  • numeric_test.py
  • pdarray_creation_test.py Conversion for new test framework #2573
  • Convert security_test.py for new test framework #2640
  • Convert segarray_test.py to new test framework #2642
  • series_test.py Conversion for new test framework #2651
  • sort_test.py Conversion for new test framework #2654
  • stats_test.py Conversion for new test framework #2656
  • string_test.py conversion for new test framework #2700
  • where_test.py refactor for new testing framework #2705
  • io_util_test.py reformat for new test framework #2668
    Complex Conversions
  • categorical_test.py conversion for new test framework #2686
  • client_test.py conversion for new test framework #2538
  • coargsort_test.py conversion for new test framework #2625
  • dataframe_test.py Conversion for new test framework #2559
  • datetime_test.py conversion to new framework #2697
  • extrema_test.py conversion for new test framework #2694
  • groupby_test.py Conversion to new test framework #2607
  • io_test.py/parquet_test.py/import_export_test.py conversion for new test framework #2539
  • message_test.py Conversion for new test framework #2542
  • nan_test.py conversion to new test framework #2610
  • operator_test.py conversion to new framework #2688
  • regex_test.py conversion to new test framework #2659
  • setops_test.py Conversion to new test framework #2547
  • symbol_table_test.py refactor to new test framework #2684

  check.py should be reviewed to determine if any testing needs to be moved to another file. Otherwise, it will be removed.

• Rename proto-tests and make commands #3669

• Configure benchmarks so that testing and benchmarks use same configuration files

• Verify that all test files pass flake8. Correct any that do not. #2768

• Ensure CI is functional. Add flake8 check to CI for tests

[stress-tess]

maybe we add create dashboards to display JSON output? im not sure about that one which is why i didn't edit, but the rest looks like the right way forward to me!

[Ethan-DeBandi99]

Just leaving some of the notes on how to adjust configuration for this to be tested and not interfere with the existing test code. This may change as we continue through development. We want to be able to maintain the existing framework until the new suite is ready to replace it, but do not want to cause any issues between them.

To run with the Prototype Test Suite:
This will allow make test and python3 -m pytest to leverage the prototype code for testing.

• Rename tests directory to OLD_tests
• Rename PROTO_tests directory to tests
• Rename pytest.ini to pytest_OLD.ini
• Rename pytest_PROTO.ini to pytest.ini

To revert before push so existing test suite runs:

• Rename tests directory to PROTO_tests
• Rename OLD_tests directory to tests
• Rename pytest.ini to pytest_PROTO.ini
• Rename pytest_OLD.ini to pytest.ini

Additional Notes
The following modifications will not interfere with functionality, but prevent additional changes from being required when switching between suites. These should be removed when the Prototype suite is released, but will not negatively impact anything if they are not.

• In pytest_PROTO.ini under norecursedirs OLD_tests and benchmark* this will prevent the conftest.py files in these directories from interfering with incremental changes to the prototype.
• In pytest.ini under norecursedirs, PROTO_tests will prevent the conftest.py from being read and interfering with existing tests.

[Ethan-DeBandi99]

@pierce314159, @hokiegeek2 - Wanted to get your input on how we should handle problem size because I think there are a few ways we can go about it. First, I don't think that we need the problem applied to every test case, but for example for IO we definitely need to be able to adjust the problem size. I have laid out some options below.

Option 1
Problem size is configurable as a single integer value. This is what we currently use for benchmarking, and in that situation I think it makes sense because we want to run a large scale test for performance measurements, but are not necessarily worried about testing edge cases.

Option 2
Configure the parser to allow the problem size to be problem size to be a comma delimited list of sizes to test with. We can then use the pytest parameterize feature to run any test with variable problem size for each of the supplied problem sizes. This is then recorded as individual test runs for each size. This would still allow for a single test size to be provide as well if not doing multi-locale testing. For anything that only needs to run a single size we could just grab a value from the list. Although, we could parameterize anything using the problem size for ultimate flexibilty.

Option 3
This would use a scaled problem size. So the input would be similar to Option 1, but the configuration code would then take the provided value and apply a multiplier. This would be a little less exact especially when testing features that are susceptible to variance with data distribution.

Option 4
For tests that we know we need to test with varying problem set sizes, we can hardcode sizes to use and ignore the problem size provided in the configuration.

Personally, I am leaning towards option 2 because it will give us ultimate flexibility and the parameters are recorded in the JSON output automatically so we will not need to add code to add the meta data to the JSON output. I wanted to get input before putting the initial architecture PR up so I can adjust a few tests to reflect the decision here.

[hokiegeek2]

@Ethan-DeBandi99 I also vote option 2

[Ethan-DeBandi99]

I set up the tests currently using the problem size from the configuration to use Option 2. Verified that it has the desired outcome. I do want to note that we will need to update how the benchmarks are using the problem size when we integrate them with the updated tests to use the same configuration.

[stress-tess]

Just noting when I run make test I always see the following error, which based on googling seems like pytest is unable to find any tests to run

make: *** [test-python] Error 5

If I update this line in pytest_PROTO.ini to just be tests/*_test, it works just fine

arkouda/pytest_PROTO.ini

Line 9 in cdd15d3

tests/tests/*_test

This is pretty weird because it needs to be a different value of my laptop than ethans. It's not pressing since we can all run the tests, but we should def figure this out at some point

[Ethan-DeBandi99]

@pierce314159 I am starting to see the same issue. Let's keep an eye on it and if it continues, we can update.

[stress-tess]

This is a future work idea, but I wanted to capture it so i don't forget when i come back

I think eventually a nice thing way to do at least edge case testing if to have a big dictionary that uses a str of the objtype/dtype as the key and value is the edge case arkouda object. This would live somewhere accessible by all the test files. This is an oversimplification but I think this could work pretty well with some tweaking

If we wanna grab a dictionary of edge cases with only certain dtypes, we could pass in those column names

edge_case_dict = global_edge_case_dict[dtypes]

and if we want only arrays of each edge case, we could do something like

edge_case_arr = global_edge_case_dict[dtype]

[ajpotts]

Replaced by the milestone: https://github.com/Bears-R-Us/arkouda/milestone/9