Small bug fix for handling binary files #131
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
hernot
pushed a commit
to hernot/hickle
that referenced
this pull request
Dec 1, 2020
…moved from the file mode. (telegraphic#131)
ezekial4
pushed a commit
to ezekial4/hickle
that referenced
this pull request
Dec 14, 2020
* no copy when loading astropy * typo * Bumping to 3.4.4 * Package license and requirements files * Bump to 3.4.5 * Added some tests to Travis CI and AppVeyor that check if hickle can be properly packaged up, distributed and installed. * And maybe I should not forget twine in the requirements_test.txt. * Moved the tests directory from root to ./hickle. * Added missing tests folder to the MANIFEST.in file. * Add dict-type permanently to types_dict. * Subclasses of supported types can now be pickled as well (although not yet with their proper type). * Removed all cases of saving dtypes as a single element list. * Renamed the 'type' attribute to 'base_type' in preparation for adding subclass support. * Also make sure that strings are saved as single elements. * The types_dict now uses tuples with the create-functions and the hkl_dtype key. * All create-functions now take an extra 'base_type' string, that describes what the hkl_type is that will be saved to HDF5. * Groups also obtain their base_type from the create_dataset_lookup()-function now. * The actual type of a hickled object is now saved as well (in pickled form). * Finalized implementing support for subclasses. * Coveralls -> codecov.io * Add codecov.io badge * The order of the dict item keys are now saved as well. Any dict that is loaded will be initialized with the items sorted in that order. For all types that derive from dict, the dict will be initialized using its true type directly (apparently, I had written it that way before already for some reason). This fixes telegraphic#65. * Hickle's required HDF5 attributes are now solely applied to the data group that contains the hickled Python object, instead of the entire file (this allows for hickled objects to be more easily added to already existing HDF5-files, without messing up the root of that file). * Datasets and groups now solely use 'group_xxx' if there is more than a single item at that level. All 'create_xxx' functions are now passed a string containing the name of the group/dataset that they should create. * Added forgotten base_key_type attribute to PyContainer. * Reverted working tree back to before using 'track_order'. Added missing 'six' requirement. Added a test for testing the dumping and loading of an OrderedDict. * The root of a hickled group is no longer read now, as it is not necessary. Removed the auxiliary attributes that were required for reading it. The true type of a dict key is now saved as well, even though it is not used for anything (simply saving it now in case we want to use it later). * The version is now stored using a single Python file, whose strings are read using regex. * HDF5 groups can now be given as a file_obj as well when dumping and loading. Providing a path that does not start with '/' will automatically add it to it. Added tests for these functionalities. * Arbitrary-precision integers can now be dumped and loaded properly. * Also make sure that 'long' dtypes are taken into account on Python 2. * make hickle work with pathlib.Path Basically, any package /module that saves to file supports this too (including `h5py`). * make Python 2 compatible * Changed wording. * Added six requirement, and added minimum versions for all requirements. * Now 'dill' is always used for dumping and loading serialized objects. Added a test for dumping and loading a local session function. * Add support for actually loading serialized data in Python 2. * Add new test to main as well. * Make sure new changes are also used for Python 2 * Update file_opener re telegraphic#123 * Fixed documentation of `dump` and `load` to be NumPy doc style (and look a bit better). Replaced broken `pickle` documentation link with a proper one. * Only lists and tuples are now recognized as acceptable iterables. All other iterables are either handled separately, or simply pickled. * Changed the lookup system to use object typing for more consistency. * Added test for detecting the problem raised in telegraphic#125 * Added support for hickling dicts with slashes in their dict keys. * Make sure that functional backslashes still work properly in dict keys. * Loaders are now only loaded when they are required for dumping or loading a specific object. * Make sure to do proper future import. * Raise an error if a dict item key contains a double backslash. * Only filter out import errors due to the loader not being found. * As Python 2 apparently only reports the last part of the name of a non-importable module, search for something a bit more specific. * Some small QoL changes. * The py_type of a pickled object is no longer saved to HDF5, as it is not necessary to restore the object's original state. * Removed legacy support for v1 and v2. Added start of legacy support for v3. v4 now stores its HDF5 attributes using a 'HICKLE_' prefix, to allow users to add attributes to the group without interference. * Objects can now be hickled to existing HDF5 groups, as long as they don't contain any datasets or groups themselves. * Made sure that v3 always uses relative imports, to make it easier to port any functionality change from v3 to v4. (Even though I am not a fan of relative imports) * The version is now stored using a single Python file, whose strings are read using regex. * Backported change to version store location to v3 as well. Bumped version to 3.4.7 to include the latest changes. * Removed support for Python 2. Added legacy support for hickle v3 (currently uses v3.4.7). * Remove testing for Python 2.7 as well. * Always specify the mode with which to open an HDF5-file. * Test requirements updates. * And make sure to always execute 'pytest'. * Removed basically everything that has to do with Python 2.7 from v4. As legacy_v3 is expected to be able to load files made with Python 2.7, these are not changed. * Many, many QoL changes. Converted all v4 files to be PEP8 compliant. Rewritten 'load_python3' into 'load_builtins'. The 'load_numpy' and 'load_builtins' modules are only loaded when they are required, like all other loaders. Removed the 'containers_type_dict' as the true container type is already saved anyway. Astropy's classes are now reinitialized using their true type instantly. Astropy constants can now be properly dumped and loaded. * Save types of dict keys as a normal string as well. * Some minor improvements. * Added test for opening binary files, and make sure that any 'b' is removed from the file mode. (telegraphic#131) * Added pytests for all uncovered lines. Removed lines that are never used (and thus cannot be covered). Added 'NoneType' to the dict of acceptable dict key types, as Nones can be used as keys. * Replaced all instances of 'a HDF' with 'an HDF'. * Removed the index.md file in the docs, as it is now simply pointing to the README.md file instead. * Badges! * Added few classifiers to the setup.py file. * Update requirement for pytest. * Removed use of 'track_times'. * NumPy arrays with unicode strings can now be properly dumped and loaded. * NumPy arrays containing non-NumPy objects can now be properly dumped and loaded. * Added all missing tests for full 100% coverage!!! * Make sure that kwargs is always passed to 'create_dataset'. * If hickle fails to save an object to HDF5 using its standard methods, it will fall back to pickling it (and emits a warning saying why that was necessary). Simplified the way in which python scalars are hickled. * Actually mention that the line in parentheses is the reason for serializing. * Use proper development status identifier. * Make sure to preserve the subclass type of a NumPy array when dumping. * make sure that a SkyCoord object will be properly saved and retrieved when it's a scalar or N-D array * revert the change to legacy_v3/load_astropy.py * Updated legacy v3 to v3.4.8. Co-authored-by: Kewei Li <kewl@microsoft.com> Co-authored-by: Danny Price <dan@thetelegraphic.com> Co-authored-by: Isuru Fernando <isuruf@gmail.com> Co-authored-by: Ellert van der Velden <ellert_vandervelden@outlook.com> Co-authored-by: Bas Nijholt <basnijholt@gmail.com> Co-authored-by: Rui Xue <rx.astro@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.