.pickle files are binary files and not inherently secure. They therefore should be best included elsewhere in PyBaMM's packaging infrastructure when publishing to common distribution channels like PyPI owing to security reasons in the case where these binaries are modified with malicious intent and make it to the develop branch or a PyPI release where PyBaMM can be downloaded from (binary or non-binary). This means they should neither be included in the source distribution (controlled by MANIFEST.in), nor in the wheels (in the case where the wheel is built from the sdist). Binary platform-specific wheels for PyBaMM cannot be built from sdists right now and have to be built in-tree (#3651, #3564), and require restructuring the build system configuration at least, if not rewriting what files are copied into the package structure – since pybamm/input/comsol_results/ is a namespace package (via setuptools.command.install_data).

We have the following options:

1. Store in JSON, CSV, or TXT file formats (non-binary) instead of .pickle
2. Use pooch to reproducibly download these files when needed and adjust PyBaMM's public API plus example notebooks for this, while moving these files to a separate repository in the pybamm-team organisation, and marking the repository as an archive to prevent tampering.
3. Use non-binary storage solutions from xarray or zarr because they can handle multi-dimensional data

Out of these, point 2 requires an internet connection and those wishing to load COMSOL results from these files would need access to the internet to download them, and therefore should be the last option to consider.

Task list, as of 10/05/2024

There are two major things to be done:

• Convert pickle files to non-binary files (done in Using JSON to import comsol results  #4064)
• Use https://github.com/fatiando/pooch as a required dependency to fetch these files plus adjust for the changes here
  • Set up a new repository: https://github.com/pybamm-team/pybamm-data
    • Add a README and some documentation
    • Copy the files and delete them from the PyBaMM main (this) repository (and additionally store checksums in a text file somewhere), and publish a GitHub Release there to store them perhaps (shorter URL)
  • Refer to the Scientific Python guide: https://learn.scientific-python.org/development/patterns/data-files/#downloading-larger-files-on-demand
  • Use pytest's conditional markers or regular markers to skip tests where access to the internet is not available (adjust noxfile.py, test-related files, etc.)
  • [z] Ensure files are not copied into the sdist (control MANIFEST.in and setuptools-specific package data)
  • Unsure about this point – maybe provide a public API to fetch these files (with the URLs kept hardcoded)?