Merge pull request #144 from sciris/fixes-and-updates

Fixes and updates

.github/workflows/tests.yaml

@@ -0,0 +1,33 @@
+name: Sciris CI workflow
+on: [pull_request]
+
+jobs:
+  install_and_test:
+    runs-on: ubuntu-latest
+    timeout-minutes: 5
+    strategy:
+      fail-fast: false
+      max-parallel: 8
+      matrix:
+        python-version: [ '3.8' ]
+    name: Install and test
+    steps:
+      - name: Checkout sources
+        uses: actions/checkout@v1
+      - uses: actions/setup-python@master
+        with:
+          python-version: ${{ matrix.python-version }}
+          architecture: x64
+      - name: Install Sciris
+        run: python setup.py develop
+      - name: Install tests
+        run: pip install pytest
+      - name: Run API tests
+        working-directory: ./tests
+        run: pytest test_*.py --durations=0 --junitxml=test-results.xml # Run actual tests
+      - name: Upload test results
+        uses: actions/upload-artifact@v1
+        if: failure()
+        with:
+          name: test_results
+          path: tests/test-results.xml

CHANGELOG.rst

@@ -0,0 +1,111 @@
+What's new
+==========
+
+All notable changes to this project will be documented in this file.
+
+By import convention, components of the Sciris library are listed beginning with ``sc.``, e.g. ``sc.odict()``.
+
+
+Version 1.0.0 (2020-11-30)
+--------------------------
+This major update (and official release!) includes many new utilities adopted from the `Covasim <http://covasim.org>`__ and `Atomica <http://atomica.tools>`__ libraries, as well as important improvements and bugfixes for parallel processing, object representation, and file I/O.
+
+New functions
+~~~~~~~~~~~~~
+
+Math functions
+^^^^^^^^^^^^^^
+1. ``sc.findfirst()`` and ``sc.findlast()`` return the first and last indices, respectively, of what ``sc.findinds()`` would return. These keywords (``first`` and ``last``) can also be passed directly to ``sc.findinds()``.
+2. ``sc.randround()`` probabilistically rounds numbers to the nearest integer; e.g. 1.2 will round down 80% of the time.
+3. ``sc.cat()`` is a generalization of ``np.append()``/``np.concatenate()`` that handles arbitrary types and numbers of inputs.
+4. ``sc.isarray()`` checks if the object is a Numpy array.
+
+Plotting functions
+^^^^^^^^^^^^^^^^^^
+1. A new diverging colormap, ``'orangeblue'``, has been added (courtesy Prashanth Selvaraj). It is rather pretty; you should try it out.
+2. ``sc.get_rows_cols()`` solves the small but annoying issue of trying to figure out how many rows and columns you need to plot *N* axes. It is similar to ``np.unravel_index()``, but allows the desired aspect ratio to be varied.
+3. ``sc.maximize()`` maximizes the current figure window.
+
+Date functions
+^^^^^^^^^^^^^^
+1. ``sc.date()`` will convert practically anything to a date.
+2. ``sc.day()`` will convert practically anything to an integer number of days from a starting point; for example, ``sc.day(sc.now())`` returns the number of days since Jan. 1st.
+3. ``sc.daydiff()`` computes the number of days between two or more start and end dates.
+4. ``sc.daterange()`` returns a list of date strings or date objects between the start and end dates.
+5. ``sc.datetoyear()`` converts a date to a decimal year (from Romesh Abeysuriya via Atomica).
+
+Other functions
+^^^^^^^^^^^^^^^
+1. The "flagship" functions ``sc.loadobj()``/``sc.saveobj()`` now have shorter aliases: ``sc.load()``/``sc.save()``. These functions can be used interchangeably.
+2. A convenience function, ``sc.toctic()``, has been added that does ``sc.toc(); sc.tic()``, i.e. for sequentially timing multiple blocks of code.
+3. ``sc.checkram()`` reports the current process' RAM usage at the current moment in time; useful for debugging memory leaks.
+4. ``sc.getcaller()`` returns the name and line number of the calling function; useful for logging and version control purposes.
+5. ``sc.nestedloop()`` iterates over lists in the specified order (from Romesh Abeysuriya via Atomica).
+6. ``sc.parallel_progress()`` runs a function in parallel whilst displaying a single progress bar across all processes (from Romesh Abeysuriya via Atomica).
+7. An experimental function, ``sc.asobj()``, has been added that lets any dictionary-like object be used with attributes instead (i.e. ``foo.bar`` instead of ``foo['bar']``).
+
+Bugfixes and other improvements
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+1. ``sc.parallelize()`` now uses the ``multiprocess`` library instead of ``multiprocessing``. This update fixes bugs with trying to run parallel processing in certain environments (e.g., in Jupyter notebooks). This function also returns a more helpful error message when running in the wrong context on Windows.
+2. ``sc.prepr()`` has been updated to use a simpler method of parsing objects for display; this should be faster and more robust. A default 3 second time limit has also been added.
+3. ``sc.savejson()`` now uses an indent of 2 by default, leading to much more human-readable JSON files.
+4. ``sc.gitinfo()`` has been updated to use the code from Atomica's ``fast_gitinfo()`` instead (courtesy Romesh Abeysuriya).
+5. ``sc.thisdir()`` now no longer requires the ``__file__`` argument to be supplied to get the current folder.
+6. ``sc.readdate()`` can now handle a list of dates.
+7. ``sc.getfilelist()`` now has more options, such as to return the absolute path or no path, as well as handling file matching patterns more flexibly.
+8. ``sc.Failed`` and ``sc.Empty``, which may be encountered when loading a corrupted pickle file, are now exposed to the user (before they could only be accessed via ``sc.sc_fileio.Failed``).
+9. ``sc.perturb()`` can now use either uniform or normal perturbations via the ``normal`` argument.
+
+Renamed/removed functions
+~~~~~~~~~~~~~~~~~~~~~~~~~
+1. The function ``sc.quantile()`` has been removed. Please use ``np.quantile()`` instead (though admittedly, it is extremely unlikely you were using it to begin with).
+2. The function ``sc.scaleratio()`` has been renamed ``sc.normsum()``, since it normalizes an array by the sum.
+
+Other updates
+~~~~~~~~~~~~~
+1. Module imports were moved to inside functions, improving Sciris loading time by roughly 30%.
+2. All tests were refactored to be in consistent format, increasing test coverage by roughly 50%.
+3. Continuous integration testing was updated to use GitHub Actions instead of Travis/Tox.
+
+
+Version 0.17.4 (2020-08-11)
+---------------------------
+1. ``sc.profile()`` and ``sc.mprofile()`` now return the line profiler instance for later use (e.g., to extract additional statistics).
+2. ``sc.prepr()`` (also used in ``sc.prettyobj()``) can now support objects with slots instead of dicts.
+
+
+Version 0.17.3 (2020-07-21)
+---------------------------
+1. ``sc.parallelize()`` now explicitly deep-copies objects, since on some platforms this copying does not take place as part of the parallelization process.
+
+
+Version 0.17.2 (2020-07-13)
+---------------------------
+1. ``sc.search()`` is a new function to find nested attributes/keys within objects or dictionaries.
+
+
+Version 0.17.1 (2020-07-07)
+---------------------------
+1. ``sc.Blobject`` has been modified to allow more flexibility with saving (e.g., ``Path`` objects).
+
+
+Version 0.17.0 (2020-04-27)
+---------------------------
+1. ``sc.mprofile()`` has been added, which does memory profiling just like ``sc.profile()``.
+2. ``sc.progressbar()`` has been added, which prints a progress bar.
+3. ``sc.jsonpickle()`` and ``sc.jsonunpickle()`` have been added, wrapping the module of the same name, to convert arbitrary objects to JSON.
+4. ``sc.jsonify()`` checks objects for a ``to_json()`` method, handling e.g Pandas dataframes, and falls back to ``sc.jsonpickle()`` instead of raising an exception for unknown object types.
+5. ``sc.suggest()`` now uses ``jellyfish`` instead of ``python-levenshtein`` for fuzzy string matching.
+6. ``sc.saveobj()`` now uses protocol 4 instead of the latest by default, to avoid backwards incompatibility issues caused by using protocol 5 (only compatible with Python 3.8).
+7. ``sc.odict()`` and related classes now raise ``sc.KeyNotFoundError`` exceptions. These are derived from ``KeyError``, but fix a `bug in the string representation <https://stackoverflow.com/questions/34051333/strange-error-message-printed-out-for-keyerror>`__ to allow multi-line error messages.
+8. Rewrote all tests to be pytest-compatible.
+
+
+Version 0.16.8 (2020-04-11)
+---------------------------
+1. ``sc.makefilepath()`` now has a ``checkexists`` flag, which will optionally raise an exception if the file does (or doesn't) exist.
+2. ``sc.sanitizejson()`` now handles ``datetime.date`` and ``datetime.time``.
+3. ``sc.uuid()`` and ``sc.fast_uuid()`` now work with non-integer inputs, e.g., ``sc.uuid(n=10e3)``.
+4. ``sc.thisdir()`` now accepts additional arguments, so can be used to form a full path, e.g. ``sc.thisdir(__file__, 'myfile.txt')``.
+5. ``sc.checkmem()`` has better parsing of objects.
+6. ``sc.prepr()`` now lists properties of objects, and has some aesthetic improvements.

CODE_OF_CONDUCT.md → CODE_OF_CONDUCT.rst

@@ -1,6 +1,9 @@
-# Contributor Covenant Code of Conduct
+====================================
+Code of conduct
+====================================
 
-## Our Pledge
+Our pledge
+==========
 
 In the interest of fostering an open and welcoming environment, we as
 contributors and maintainers pledge to making participation in our project and
@@ -9,29 +12,31 @@ size, disability, ethnicity, sex characteristics, gender identity and expression
 level of experience, education, socio-economic status, nationality, personal
 appearance, race, religion, or sexual identity and orientation.
 
-## Our Standards
+Our standards
+=============
 
 Examples of behavior that contributes to creating a positive environment
 include:
 
-* Using welcoming and inclusive language
-* Being respectful of differing viewpoints and experiences
-* Gracefully accepting constructive criticism
-* Focusing on what is best for the community
-* Showing empathy towards other community members
+*   Using welcoming and inclusive language
+*   Being respectful of differing viewpoints and experiences
+*   Gracefully accepting constructive criticism
+*   Focusing on what is best for the community
+*   Showing empathy towards other community members
 
 Examples of unacceptable behavior by participants include:
 
-* The use of sexualized language or imagery and unwelcome sexual attention or
- advances
-* Trolling, insulting/derogatory comments, and personal or political attacks
-* Public or private harassment
-* Publishing others' private information, such as a physical or electronic
- address, without explicit permission
-* Other conduct which could reasonably be considered inappropriate in a
- professional setting
+*   The use of sexualized language or imagery and unwelcome sexual attention or
+    advances
+*   Trolling, insulting/derogatory comments, and personal or political attacks
+*   Public or private harassment
+*   Publishing others' private information, such as a physical or electronic
+    address, without explicit permission
+*   Other conduct which could reasonably be considered inappropriate in a
+    professional setting
 
-## Our Responsibilities
+Our responsibilities
+====================
 
 Project maintainers are responsible for clarifying the standards of acceptable
 behavior and are expected to take appropriate and fair corrective action in
@@ -43,7 +48,8 @@ that are not aligned to this Code of Conduct, or to ban temporarily or
 permanently any contributor for other behaviors that they deem inappropriate,
 threatening, offensive, or harmful.
 
-## Scope
+Scope
+=====
 
 This Code of Conduct applies both within project spaces and in public spaces
 when an individual is representing the project or its community. Examples of
@@ -52,10 +58,11 @@ address, posting via an official social media account, or acting as an appointed
 representative at an online or offline event. Representation of a project may be
 further defined and clarified by project maintainers.
 
-## Enforcement
+Enforcement
+===========
 
 Instances of abusive, harassing, or otherwise unacceptable behavior may be
-reported by contacting the project team at info@sciris.org. All
+reported by contacting the project team at covasim@idmod.org. All
 complaints will be reviewed and investigated and will result in a response that
 is deemed necessary and appropriate to the circumstances. The project team is
 obligated to maintain confidentiality with regard to the reporter of an incident.
@@ -65,12 +72,14 @@ Project maintainers who do not follow or enforce the Code of Conduct in good
 faith may face temporary or permanent repercussions as determined by other
 members of the project's leadership.
 
-## Attribution
+Attribution
+===========
 
-This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4,
-available at https://www.contributor-covenant.org/version/1/4/code-of-conduct.html
+This Code of Conduct is adapted from the `Contributor Covenant`_, version 1.4,
+available at https://www.contributor-covenant.org/version/1/4/code-of-conduct.html.
 
-[homepage]: https://www.contributor-covenant.org
+.. _Contributor Covenant: https://www.contributor-covenant.org
 
-For answers to common questions about this code of conduct, see
-https://www.contributor-covenant.org/faq
+For answers to common questions about this code of conduct, see the FAQ_.
+
+.. _FAQ: https://www.contributor-covenant.org/faq