Release of v4.0.0? #117

1313e · 2019-10-09T03:22:00Z

Also create this PR in advance, such that we can keep track of what issues have already been dealt with. This includes changes made in any (non-merged) PR to the dev branch.

Issues solved:

OrderedDict is now supported (fixes Add support for OrderedDict #65);
data_0 is no longer used if there is only a single data group/set (fixes Redundant data_0? #44);
HDF5 groups can now be dumped to and loaded from (fixes Dump objects into h5py.Group #54);
Integers using Python's arbitrary-precision (integers larger than 64-bit) can now be dumped and loaded properly (fixes Can't dump long integers #113);
Replaced broken link to pickle documentation with proper one (fixes Documentation example lead to 404 web page #122);
Objects that appear to be iterable are no longer considered as such unless hickle knows for sure they are iterables (fixes cannot hickle simple class #70 and fixes support for python copy protocol __setstate__ __getstate__ if present in object #125);
Dict keys with slashes are now supported (fixes Feature proposal: dealing with slashes in dict keys #124);
Loaders are only loaded when they are required for dumping or loading specific objects (fixes Long import times of hickle #114);

NOTE: Before this gets merged, a v3 branch must be made of master.

…t yet with their proper type).

… subclass support.

…dtype key.

…ibes what the hkl_type is that will be saved to HDF5.

…unction now.

…form).

Any dict that is loaded will be initialized with the items sorted in that order. For all types that derive from dict, the dict will be initialized using its true type directly (apparently, I had written it that way before already for some reason). This fixes #65.

…group that contains the hickled Python object, instead of the entire file (this allows for hickled objects to be more easily added to already existing HDF5-files, without messing up the root of that file).

…a single item at that level. All 'create_xxx' functions are now passed a string containing the name of the group/dataset that they should create.

Added missing 'six' requirement. Added a test for testing the dumping and loading of an OrderedDict.

…sary. Removed the auxiliary attributes that were required for reading it. The true type of a dict key is now saved as well, even though it is not used for anything (simply saving it now in case we want to use it later).

Merge master -> dev

Hickle subclasses

…re read using regex.

codecov · 2019-10-09T03:39:46Z

Codecov Report

Merging #117 into master will increase coverage by 10.24%.
The diff coverage is 96.78%.

@@             Coverage Diff             @@
##           master     #117       +/-   ##
===========================================
+ Coverage   87.40%   97.65%   +10.24%     
===========================================
  Files          12        9        -3     
  Lines         762      596      -166     
===========================================
- Hits          666      582       -84     
+ Misses         96       14       -82

Impacted Files	Coverage Δ
hickle/hickle.py	`93.57% <89.92%> (+10.30%)`	⬆️
hickle/__init__.py	`100.00% <100.00%> (ø)`
hickle/__version__.py	`100.00% <100.00%> (ø)`
hickle/helpers.py	`100.00% <100.00%> (+4.16%)`	⬆️
hickle/loaders/load_astropy.py	`100.00% <100.00%> (+15.50%)`	⬆️
hickle/loaders/load_builtins.py	`100.00% <100.00%> (ø)`
hickle/loaders/load_numpy.py	`100.00% <100.00%> (+11.53%)`	⬆️
hickle/loaders/load_scipy.py	`100.00% <100.00%> (ø)`
hickle/lookup.py	`100.00% <100.00%> (+11.94%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ed2b19b...77f29ad. Read the comment docs.

…oading. Providing a path that does not start with '/' will automatically add it to it. Added tests for these functionalities.

1313e · 2020-06-03T04:48:12Z

@telegraphic I am currently planning on continuing with the rewrite for v4.0.0 this weekend (I will therefore also be renaming and moving some branches around).
Do you agree with me that v4 should be released rather soon, given the several improvements it brings (which are basically waiting for like 8 or 9 months now)?
If so, I would like your opinion on dealing with legacy versions: Should we keep supporting all hickle versions, or only those made with v3?

…not necessary to restore the object's original state.

Added start of legacy support for v3. v4 now stores its HDF5 attributes using a 'HICKLE_' prefix, to allow users to add attributes to the group without interference.

…on't contain any datasets or groups themselves.

Added legacy support for hickle v3 (currently uses v3.4.7).

As legacy_v3 is expected to be able to load files made with Python 2.7, these are not changed.

Converted all v4 files to be PEP8 compliant. Rewritten 'load_python3' into 'load_builtins'. The 'load_numpy' and 'load_builtins' modules are only loaded when they are required, like all other loaders. Removed the 'containers_type_dict' as the true container type is already saved anyway. Astropy's classes are now reinitialized using their true type instantly. Astropy constants can now be properly dumped and loaded.

More changes for v4

1313e · 2020-06-10T13:10:49Z

@telegraphic Can you btw make a new release, as a legacy hickle file of v3.4.7 needs to be added to v4.
Also, would you like me to do some more code cleaning (like, checking if all documentation and comments is up-to-date; remove unnecessary lines; etc.)?

Removed lines that are never used (and thus cannot be covered). Added 'NoneType' to the dict of acceptable dict key types, as Nones can be used as keys.

…o the README.md file instead.

1313e · 2020-06-19T05:23:48Z

@telegraphic Any chances we can get this done this weekend?
Currently, the only things that are remaining are (in this order):

Solve Hickling NumPy arrays with dtypes that are not supported by h5py #90 (I actually just got an idea on how to fix this);
Make a decision on 'track_times' argument for 'dump' has no use #130;
Release v3.4.7 (as I need to make a legacy v3 file for v4 testing);
Migrate hickle from travis.org to travis.com;
Merge Some more clean-up and adding of missing tests #132 (which then should contain the first two things as well);
Review all the changes in this PR;
Make a v3 branch based on master (for future maintenance by either us or someone else, as we are dropping Python 2.7 support);
Merge this PR into master;
Release v4.0.0.

Unless you have something that you would like to add (or like me to add) for v4, I am pretty much done I think.

Some more clean-up and adding of missing tests

telegraphic

Downloaded and ran tests, LGTM

telegraphic · 2020-06-19T12:38:04Z

@telegraphic Any chances we can get this done this weekend?
Currently, the only things that are remaining are (in this order):
* Solve #90 (I actually just got an idea on how to fix this);

Will leave this one to you

* Make a decision on #130;

Agreed, we can dump the track_times as kwargs will catch it

* Release v3.4.7 (as I need to make a legacy v3 file for v4 testing);

Just released on pypi

* Migrate `hickle` from travis.org to travis.com;

I think I've done this (migration is in beta)

* Merge #132 (which then should contain the first two things as well);

Merged

* Review all the changes in this PR;

Done

* Make a `v3` branch based on `master` (for future maintenance by either us or someone else, as we are dropping Python 2.7 support);

Makes sense, v3 branch made

* Merge this PR into `master`;

About to do so

* Release v4.0.0.
Unless you have something that you would like to add (or like me to add) for v4, I am pretty much done I think.

Fantastic stuff
!

1313e · 2020-06-19T13:10:20Z

@telegraphic You may want to wait with releasing v4 until I have removed the track_times kwargs.

telegraphic · 2020-06-19T13:39:27Z

was a bit trigger happy, yes will hold off.

1313e · 2020-06-19T14:09:03Z

was a bit trigger happy, yes will hold off.

A little bit.

Release of v4.0.0

1313e and others added 19 commits September 14, 2019 14:22

Add dict-type permanently to types_dict.

66811fc

Subclasses of supported types can now be pickled as well (although no…

98d7be5

…t yet with their proper type).

Removed all cases of saving dtypes as a single element list.

677905f

Renamed the 'type' attribute to 'base_type' in preparation for adding…

9fb915f

… subclass support.

Also make sure that strings are saved as single elements.

0797eb3

The types_dict now uses tuples with the create-functions and the hkl_…

ec67921

…dtype key.

All create-functions now take an extra 'base_type' string, that descr…

e907c5c

…ibes what the hkl_type is that will be saved to HDF5.

Groups also obtain their base_type from the create_dataset_lookup()-f…

f1b76aa

…unction now.

The actual type of a hickled object is now saved as well (in pickled …

bb4252a

…form).

Finalized implementing support for subclasses.

24bcbdb

Hickle's required HDF5 attributes are now solely applied to the data …

b96cb09

…group that contains the hickled Python object, instead of the entire file (this allows for hickled objects to be more easily added to already existing HDF5-files, without messing up the root of that file).

Datasets and groups now solely use 'group_xxx' if there is more than …

ca80fa8

…a single item at that level. All 'create_xxx' functions are now passed a string containing the name of the group/dataset that they should create.

Added forgotten base_key_type attribute to PyContainer.

81a667a

Reverted working tree back to before using 'track_order'.

09577ac

Added missing 'six' requirement. Added a test for testing the dumping and loading of an OrderedDict.

Merge pull request #115 from telegraphic/master

0927148

Merge master -> dev

Merge pull request #112 from 1313e/hickle_subclasses

e3eb919

Hickle subclasses

The version is now stored using a single Python file, whose strings a…

ffc09d8

…re read using regex.

1313e added 4 commits October 9, 2019 14:45

HDF5 groups can now be given as a file_obj as well when dumping and l…

5e7c0ec

…oading. Providing a path that does not start with '/' will automatically add it to it. Added tests for these functionalities.

Arbitrary-precision integers can now be dumped and loaded properly.

0e54212

Also make sure that 'long' dtypes are taken into account on Python 2.

55c26a1

Changed wording.

992358e

1313e mentioned this pull request Apr 9, 2020

Huge dict() object loads failed #121

Closed

1313e added 4 commits April 9, 2020 15:40

Merge branch 'master' into dev

b7feac5

Merge remote-tracking branch 'remotes/origin/dev' into 1313e_dev

03e9588

Make sure new changes are also used for Python 2

94bc279

Merge branch 'dev' into 1313e_dev

450cdf0

1313e and others added 9 commits June 9, 2020 13:24

The py_type of a pickled object is no longer saved to HDF5, as it is …

3058152

…not necessary to restore the object's original state.

Removed legacy support for v1 and v2.

236cb14

Added start of legacy support for v3. v4 now stores its HDF5 attributes using a 'HICKLE_' prefix, to allow users to add attributes to the group without interference.

Objects can now be hickled to existing HDF5 groups, as long as they d…

5e04d1b

…on't contain any datasets or groups themselves.

Removed support for Python 2.

75f0057

Added legacy support for hickle v3 (currently uses v3.4.7).

Remove testing for Python 2.7 as well.

d4342af

Removed basically everything that has to do with Python 2.7 from v4.

ec81512

As legacy_v3 is expected to be able to load files made with Python 2.7, these are not changed.

Merge pull request #129 from 1313e/1313e_dev

d8c7f70

More changes for v4

Merge remote-tracking branch 'remotes/upstream/master' into dev_upstream

a3dd28d

1313e added 9 commits June 11, 2020 16:22

Save types of dict keys as a normal string as well.

ee68369

Some minor improvements.

c1167e4

Merge remote-tracking branch 'remotes/upstream/master' into dev_upstream

cfbb941

Merge remote-tracking branch 'remotes/upstream/dev' into 1313e_dev

e49780a

Added pytests for all uncovered lines.

0c9201a

Removed lines that are never used (and thus cannot be covered). Added 'NoneType' to the dict of acceptable dict key types, as Nones can be used as keys.

Replaced all instances of 'a HDF' with 'an HDF'.

8789d00

Removed the index.md file in the docs, as it is now simply pointing t…

5c6258c

…o the README.md file instead.

Badges!

8c2d4a9

Added few classifiers to the setup.py file.

b4c93ca

Merge pull request #132 from 1313e/1313e_dev

6c40505

Some more clean-up and adding of missing tests

1313e mentioned this pull request Jun 19, 2020

enhancement request: ability to export dicts as shareable/generic HDF5 #133

Closed

Merge remote-tracking branch 'remotes/upstream/master' into dev_upstream

77f29ad

telegraphic reviewed Jun 19, 2020

View reviewed changes

telegraphic merged commit 7bc9965 into master Jun 19, 2020

hernot pushed a commit to hernot/hickle that referenced this pull request Dec 1, 2020

v4.0.0 -- Merge pull request telegraphic#117 from telegraphic/dev

afdfe1a

Release of v4.0.0

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Release of v4.0.0? #117

Release of v4.0.0? #117

1313e commented Oct 9, 2019 •

edited

codecov bot commented Oct 9, 2019 •

edited

1313e commented Jun 3, 2020

1313e commented Jun 10, 2020 •

edited

1313e commented Jun 19, 2020

telegraphic left a comment

telegraphic commented Jun 19, 2020

1313e commented Jun 19, 2020

telegraphic commented Jun 19, 2020

1313e commented Jun 19, 2020

Release of v4.0.0? #117

Release of v4.0.0? #117

Conversation

1313e commented Oct 9, 2019 • edited

codecov bot commented Oct 9, 2019 • edited

Codecov Report

1313e commented Jun 3, 2020

1313e commented Jun 10, 2020 • edited

1313e commented Jun 19, 2020

telegraphic left a comment

Choose a reason for hiding this comment

telegraphic commented Jun 19, 2020

1313e commented Jun 19, 2020

telegraphic commented Jun 19, 2020

1313e commented Jun 19, 2020

1313e commented Oct 9, 2019 •

edited

codecov bot commented Oct 9, 2019 •

edited

1313e commented Jun 10, 2020 •

edited