Skip to content

Conversation

@jimmo
Copy link
Member

@jimmo jimmo commented Jun 1, 2023

This splits out each algorithm into its own extension package, so that only the necessary algorithms can be installed.

This allows for a significant reduction in RAM and flash. i.e. previously installing hashlib meant that all algorithms were imported.

Additionally ensures that any built-in hash algorithms (from uhashlib) are still exposed (e.g. md5), and retains the existing behavior to use the built-in preferentially. (Replaces #674)

Also includes a refactoring of the algorithms to reduce code size and reduce the number of allocations they do as well as using bytearrays in place of list-of-int where possible.

Add more comprehensive tests (using unittest).

This work was funded through GitHub Sponsors.

@jimmo
Copy link
Member Author

jimmo commented Jun 1, 2023

For example, if you previously only wanted, say sha512, the package would install

 317 __init__.mpy
  90 _sha224.mpy
6157 _sha256.mpy
  90 _sha384.mpy
8692 _sha512.mpy

and all of this code would be resident (15346 bytes of .mpy files total). Even if you already had built-in sha256, it still ends up importing sha256 to provide sha224.

Now you only install and load the files you need (which are also each smaller):

  321 __init__.mpy
  272 _sha224.mpy
 4889 _sha256.mpy
  367 _sha384.mpy
 6983 _sha512.mpy
  557 _sha.mpy

(Note that __init__.py and _sha.py are always required, and _sha224.py uses _sha256.py, and _sha384.py uses _sha512.py)

So now if you just want to add sha512, you can just install hashlib-sha512 and it will install and load just 7861 bytes of .mpy files.

@KipCrossing
Copy link

@jimmo What else needs doing before this can be merged?

This splits out each algorithm into its own extension package, so that only
the necessary algorithms can be installed.

This allows for a significant reduction in RAM and flash. i.e. previously
installing hashlib meant that all algorithms were imported.

Additionally ensures that any built-in hash algorithms (from uhashlib) are
still exposed (e.g. `md5`), and retains the existing behavior to use the
built-in preferentially.

Also includes a refactoring of the algorithms to reduce code size and
reduce the number of allocations they do as well as using bytearrays in
place of list-of-int where possible.

Add more comprehensive tests (using unittest).

This work was funded through GitHub Sponsors.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
@dpgeorge dpgeorge force-pushed the hashlib-builtins-v2 branch from 1635d69 to da5ddfc Compare July 20, 2023 13:28
@dpgeorge
Copy link
Member

I removed the now-empty test_hashlib.py file and rebased on master.

@dpgeorge dpgeorge merged commit da5ddfc into micropython:master Jul 20, 2023
@dpgeorge
Copy link
Member

Thanks for this, it's a very nice improvement.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants