Skip to content

Commit

Permalink
Release 0.4.1 (#368)
Browse files Browse the repository at this point in the history 
[ committed by @billschereriii ]
[ reviewed by @ashao  @mellis13  ]
  • Loading branch information
@billschereriii
billschereriii committed Jul 5, 2023
1 parent 39427e4 commit 01f56f7
Show file tree
Hide file tree
Showing 4 changed files with 79 additions and 119 deletions.
55 changes: 30 additions & 25 deletions README.md
View file
Expand Up @@ -32,12 +32,12 @@ RedisAI capabilities and include additional
features for high performance computing (HPC) applications.
SmartRedis provides clients in the following languages:

| Language | Version/Standard |
|------------|:-----------------:|
| Python | 3.7, 3.8, 3.9 |
| C++ | C++17 |
| C | C99 |
| Fortran | Fortran 2018 |
| Language | Version/Standard |
|------------|:----------------------------------------------:|
| Python | 3.7, 3.8, 3.9, 3.10 |
| C++ | C++17 |
| C | C99 |
| Fortran | Fortran 2018 (GNU/Intel), 2003 (PGI/Nvidia) |

SmartRedis is used in the [SmartSim library](https://github.com/CrayLabs/SmartSim).
SmartSim makes it easier to use common Machine Learning (ML) libraries like
Expand All @@ -57,37 +57,42 @@ part of the SmartSim documentation.

## Dependencies

SmartRedis utilizes the following libraries.
SmartRedis utilizes the following libraries:

- [NumPy](https://github.com/numpy/numpy)
- [Hiredis](https://github.com/redis/hiredis) 1.0.0
- [Redis-plus-plus](https://github.com/sewenew/redis-plus-plus) 1.2.3
- [Hiredis](https://github.com/redis/hiredis) 1.1.0
- [Redis-plus-plus](https://github.com/sewenew/redis-plus-plus) 1.3.5

## Publications

The following are public presentations or publications using SmartRedis

- [Collaboration with NCAR - CGD Seminar](https://www.youtube.com/watch?v=2e-5j427AS0)
- [Using Machine Learning in HPC Simulations - paper (pre-print)](https://arxiv.org/abs/2104.09355)
- [Using Machine Learning in HPC Simulations - paper](https://www.sciencedirect.com/science/article/pii/S1877750322001065)
- [Relexi — A scalable open source reinforcement learning framework for high-performance computing - paper](https://www.sciencedirect.com/science/article/pii/S2665963822001063)

## Cite

Please use the following citation when referencing SmartSim, SmartRedis, or any SmartSim related work.
Please use the following citation when referencing SmartSim, SmartRedis, or any SmartSim related work:


Partee et al., “Using Machine Learning at Scale in HPC Simulations with SmartSim:
An Application to Ocean Climate Modeling,” arXiv:2104.09355, Apr. 2021,
[Online]. Available: http://arxiv.org/abs/2104.09355.
Partee et al., "Using Machine Learning at scale in numerical simulations with SmartSim:
An application to ocean climate modeling",
Journal of Computational Science, Volume 62, 2022, 101707, ISSN 1877-7503.
Open Access: https://doi.org/10.1016/j.jocs.2022.101707.

### bibtex

```latex
@misc{partee2021using,
title={Using Machine Learning at Scale in HPC Simulations with SmartSim: An Application to Ocean Climate Modeling},
author={Sam Partee and Matthew Ellis and Alessandro Rigazzi and Scott Bachman and Gustavo Marques and Andrew Shao and Benjamin Robbins},
year={2021},
eprint={2104.09355},
archivePrefix={arXiv},
primaryClass={cs.CE}
}
```
@article{PARTEE2022101707,
title = {Using Machine Learning at scale in numerical simulations with SmartSim:
An application to ocean climate modeling},
journal = {Journal of Computational Science},
volume = {62},
pages = {101707},
year = {2022},
issn = {1877-7503},
doi = {https://doi.org/10.1016/j.jocs.2022.101707},
url = {https://www.sciencedirect.com/science/article/pii/S1877750322001065},
author = {Sam Partee and Matthew Ellis and Alessandro Rigazzi and Andrew E. Shao
and Scott Bachman and Gustavo Marques and Benjamin Robbins},
keywords = {Deep learning, Numerical simulation, Climate modeling, High performance computing, SmartSim},
}
28 changes: 12 additions & 16 deletions doc/changelog.rst
View file
@@ -1,13 +1,17 @@
Changelog
=========

Development branch
------------------
0.4.1
-----

To be released at some future point in time
Released on July 5, 2023

Description

This release revamps the build and test systems for SmartRedis as well as improving
compatibility with different Fortran compilers and laying the groundwork for future
support for interacting with multiple concurrent backend databases:

- Documentation improvements
- Improved compatibility of type hints with third-party software
- Added type hints to the Python interface layer
Expand All @@ -26,7 +30,7 @@ Description
- Fix the spelling of the Dataset destructor's C interface (now DeallocateDataSet)
- Update Redis++ version to 1.3.8
- Refactor third-party software dependency installation
- Add pip-install target to Makefile to automate this process going forward
- Add pip-install target to Makefile to automate this process going forward (note: this was later removed)
- Added infrastructure for multiDB support

Detailed Notes
Expand All @@ -41,18 +45,10 @@ Detailed Notes
- Major revamp to simplify use of SmartRedis test system, automating most test processes (PR356_)
- Remove debug output in pybind layer associated with put_dataset (PR352_)
- Updated to the latest version of Hiredis (1.1.0) (PR351_)
- Enable parallel build for the SmartRedis examples by moving utility Fortran code
into a small static library (PR349_)
- For the NVidia toolchain only: Replaces the assumed rank feature of F2018 used
in the Fortran client with assumed shape arrays, making it possible to compile
SmartRedis with the Nvidia toolchain. (PR346_)
- Rework the build and test system to improve maintainability of the library. There have
been several significant changes, including that Python and Fortran clients are no longer
built by defaults and that there are Make variables that customize the build process.
Please review the build documentation and ``make help`` to see all that has changed. (PR341_)
- Many Fortran routines were returning logical kind = c_bool which turns out not to be
the same default kind of most Fortran compilers. These have now been refactored so that
users need not import `iso_c_binding` in their own applications (PR340_)
- Enable parallel build for the SmartRedis examples by moving utility Fortran code into a small static library (PR349_)
- For the NVidia toolchain only: Replaces the assumed rank feature of F2018 used in the Fortran client with assumed shape arrays, making it possible to compile SmartRedis with the Nvidia toolchain. (PR346_)
- Rework the build and test system to improve maintainability of the library. There have been several significant changes, including that Python and Fortran clients are no longer built by defaults and that there are Make variables that customize the build process. Please review the build documentation and ``make help`` to see all that has changed. (PR341_)
- Many Fortran routines were returning logical kind = c_bool which turns out not to be the same default kind of most Fortran compilers. These have now been refactored so that users need not import `iso_c_binding` in their own applications (PR340_)
- Update MacOS version in CI/CD tests from 10.15 to 12.0 (PR339_)
- Correct the spelling of the C DataSet destruction interface from DeallocateeDataSet to DeallocateDataSet (PR338_)
- Updated the version of Redis++ to v1.3.8 to pull in a change that ensures the redis++.pc file properly points to the generated libraries (PR334_)
Expand Down
113 changes: 36 additions & 77 deletions doc/testing.rst
View file
Expand Up @@ -2,108 +2,67 @@
Testing
*******

The following will demonstrate how to build and run the tests for
each of the SmartSim clients.

Building the Tests
==================

Before building the tests, it is assumed that the base dependencies
for SmartRedis described in the installation instructions have already
been executed.

To build the tests, you first need to install the dependencies for
testing. To download SmartRedis related testing dependencies, run
the following:

.. code-block:: bash
make test-deps
If you wish to run tests on GPU hardware, run the following command:
To build and run all tests on the local host, run the following command in the top
level of the smartredis repository:

.. code-block:: bash
make test-deps-gpu
.. note::

The test suite is currently written to be run on CPU hardware to
test model and script executions. Testing on GPU hardware
currently requires modifications to the test suite.
make test
.. note::

The tests require:
- GCC > 5
- CMake > 3.13
- GCC >= 5
- CMake >= 3.13

Since these are usually system libraries, we do not install them
for the user.

After installing dependencies and setting up your testing environment with
``setup_test_env.sh``, all tests can be built with the following command:
You can also run tests for individual clients as follows:

.. code-block:: bash
./setup_test_env.sh
make build-tests
make test-c # run C tests
make test-fortran # run Fortran tests. Implicitly, SR_FORTRAN=ON
make test-cpp # run all C++ tests
make unit-test-cpp # run unit tests for C++
make test-py # run Python tests. Implicitly, SR_PYTHON=ON
make testpy-cov # run python tests with coverage. Implicitly, SR_PYTHON=ON SR_BUILD=COVERAGE
make testcpp-cpv # run cpp unit tests with coverage. Implicitly, SR_BUILD=COVERAGE
Starting Redis
==============
Customizing the Tests
=====================

Before running the tests, users will have to spin up a Redis
cluster instance and set the ``SSDB`` environment variable.
Several Make variables can adjust the manner in which tests are run:
- SR_BUILD: change the way that the SmartRedis library is built. (supported: Release, Debug, Coverage; default for testing is Debug)
- SR_FORTRAN: enable Fortran language build and testing (default is OFF)
- SR_PYTHON: enable Python language build and testing (default is OFF)
- SR_TEST_PORT: change the base port for Redis communication (default is 6379)
- SR_TEST_NODES: change the number of Redis shards used for testing (default is 3)
- SR_TEST_REDIS_MODE: change the type(s) of Redis servers used for testing. Supported is Clustered, Standalone, UDS; default is Clustered)
- SR_TEST_REDISAI_VER: change the version of RedisAI used for testing. (Default is v1.2.3; the parameter corresponds the the RedisAI gitHub branch name for the release)
- SR_TEST_DEVICE: change the type of device to test against. (Supported is cpu, gpu; default is cpu)
- SR_TEST_PYTEST_FLAGS: tweak flags sent to pytest when executing tests (default is -vv -s)

To spin up a local Redis cluster, use the script
in ``utils/create_cluster`` as follows:
These variables are all orthogonal. For example, to run tests for all languages against
a standalone Redis server, execute the following command:

.. code-block:: bash
cd /smartredis # navigate to the top level dir of smartredis
conda activate env # activate python env with SmartRedis requirements
source setup_test_env.sh # Setup smartredis environment
cd utils/create_cluster
python local_cluster.py # spin up Redis cluster locally
export SSDB="127.0.0.1:6379,127.0.0.1:6380,127.0.0.1:6381" # Set database location
make test SR_FORTRAN=ON SR_PYTHON=ON SR_TEST_REDIS_MODE=Standalone
# run the tests (described below)
cd utils/create_cluster
python local_cluster.py --stop # stop the Redis cluster
A similar script ``utils/create_cluster/slurm_cluster.py``
assists with launching a Redis cluster for testing on
Slurm managed machines. This script has only been tested
on a Cray XC, and it may not be portable to all machines.

Running the Tests
=================

.. note::

If you are running the tests in a new terminal from the
one used to build the tests and run the Redis cluster,
remember to load your python environment with SmartRedis
dependencies, source the ``setup_test_env.sh`` file,
and set the ``SSDB`` environment variable.

To build and run all tests, run the following command in the top
level of the smartredis repository.
Similarly, it is possible to run the tests against each type of Redis server in sequence
(all tests against a standalone Redis server, then all tests against a Clustered server,
then all tests against a standalone server with a Unix domain socket connection) via the
following command:

.. code-block:: bash
make test
make test SR_FORTRAN=ON SR_PYTHON=ON SR_TEST_REDIS_MODE=All
You can also run tests for individual clients as follows:
.. note::

.. code-block:: bash
Unix domain socket connections are not supported on MacOS. If the SmartRedis test
system detects that it is running on MacOS, it will automatically skip UDS testing.

make test-c # run C tests
make test-fortran # run Fortran tests
make test-cpp # run all C++ tests
make unit-test-cpp # run unit tests for C++
make test-py # run Python tests
make testpy-cov # run python tests with coverage
make testcpp-cpv # run cpp unit tests with coverage
2 changes: 1 addition & 1 deletion setup.cfg
View file
@@ -1,6 +1,6 @@
[metadata]
name = smartredis
version = 0.4.0
version = 0.4.1
description = RedisAI clients for SmartSim
long_description = file: README.md
long_description_content_type=text/markdown
Expand Down

0 comments on commit 01f56f7

Please sign in to comment.