WIP: Spring cleaning (Treant unification, no uuids, .datreant state) #145

dotsdl · 2017-04-24T19:27:29Z

fixes #100
fixes #101
fixes #135
fixes #65

A couple weeks ago @kain88-de and @richardjgowers and I met to discuss some changes I was considering for datreant. These changes include removal of uuids (#135), removal of the concept of different Treant types from the whole library (goes toward #100), and the replacement of a bare state file with a .datreant directory indicating a Treant.

I'd like to store tags and categories in separate files under this scheme, which will require more hacking. Basically, instead of a limb such as Tags storing all its stuff with any other limb's stuff in the same file, it can do what it needs separately. Not sure if this idea will be something we like but toying with it still.

Once the dust settles I'll also add a converter script for putting an existing Treant into the new form.

There is no state file for the overall Treant anymore. Now, what makes a Treant is the presence of the `.datreant` directory alone. Limbs make whatever files they need inside there to function.

All tests pass again, so we shouldn't have any obvious regressions. Now need to add tests that Treants are as hardy as they should be with new machinery.

kain88-de · 2017-05-08T21:15:12Z

src/datreant/core/treants.py

+
+    @property
+    def _treantfile(self):
+        return os.path.join(self.abspath, treantdir_name, treantfile_name)


return os.path.join(self._treantdir, treantfile_name)

kain88-de · 2017-05-08T21:16:51Z

src/datreant/core/treants.py

-
-        """
-        return self._backend.filename
+        return Path(self._backend.get_location()).absolute().parent


why not Path(self._treantdir).absolute().parent?

kain88-de

Looks good so far

kain88-de · 2017-05-09T08:05:57Z

src/datreant/core/names.py

@@ -0,0 +1,2 @@
+treantdir_name = '.datreant'
+treantfile_name = 'state.json'


shouldn't globals we written in all caps

These aren't tested yet, so much to do still.

dotsdl · 2017-05-28T19:27:10Z

Status update: this will be a big PR, essentially being a refresh of much of the core library. I went through the list of open issues last night, and the changes we're making here will make many of our older issues either solved or irrelevant. This is similar in effect (though not in scale) to the MDAnalysis topology system refactor, and I'm shooting for consistency throughout the library.

Hang tight; more to come. :D

codecov-io · 2017-05-28T19:32:51Z

Codecov Report

Merging #145 into develop will increase coverage by 1.93%.
The diff coverage is 80.29%.

@@             Coverage Diff             @@
##           develop     #145      +/-   ##
===========================================
+ Coverage     77.7%   79.64%   +1.93%     
===========================================
  Files           14       12       -2     
  Lines         1624     1243     -381     
  Branches       354      283      -71     
===========================================
- Hits          1262      990     -272     
+ Misses         306      206     -100     
+ Partials        56       47       -9

Impacted Files	Coverage Δ
src/datreant/core/__init__.py	`100% <ø> (ø)`	⬆️
src/datreant/core/util.py	`100% <100%> (ø)`	⬆️
src/datreant/core/exceptions.py	`100% <100%> (ø)`
src/datreant/core/names.py	`100% <100%> (ø)`
src/datreant/core/manipulators.py	`87.5% <100%> (-1.08%)`	⬇️
src/datreant/core/state.py	`71.81% <100%> (ø)`
src/datreant/core/collections.py	`67.68% <61.41%> (-0.29%)`	⬇️
src/datreant/core/trees.py	`84.31% <66.66%> (+0.38%)`	⬆️
src/datreant/core/treants.py	`74.11% <71.42%> (-9.56%)`	⬇️
src/datreant/core/metadata.py	`90.36% <94.08%> (ø)`
... and 6 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update dca253e...5a2d993. Read the comment docs.

kain88-de · 2017-06-23T19:54:34Z

@dotsdl any progress on this?

dotsdl · 2017-06-24T01:52:22Z

@kain88-de been working on-and-off on it. Have a new application to build for work that I think will be MDSynthesis-like, in that it uses Treant as a base-class for a domain-specific object. We use datreant pretty extensively for getting things done there, but I don't have a lot of spare cycles at the moment to work directly on it.

I think when @richardjgowers is here for a week I'll find some motivation to just finish it. Standby. :D

use pathlib directly

This will allow to drop the wrapper once the library is py3 only.

kain88-de · 2017-07-21T20:03:55Z

funny the code now works on python 3.6 and only that version

kain88-de · 2017-07-21T20:07:06Z

The error seems to be with the multiprocessing module. I don't have any experience with it myself. @dotsdl would be nice if you could have a lock

kain88-de · 2017-07-21T20:30:48Z

OK I fixed the python 2.7 errors I found as well. But really no clue about the multiprocessing errors. It looks like the processes never finish

jbarnoud · 2017-07-21T20:33:27Z

setup.py

@@ -32,5 +32,5 @@
      license='BSD',
      long_description=open('README.rst').read(),
      tests_require = ['numpy', 'pytest>=2.10', 'mock'],
-      install_requires=['asciitree', 'pathlib2', 'scandir', 'six', 'fuzzywuzzy']
+      install_requires=['asciitree', 'pathlib2', 'scandir', 'six', 'fuzzywuzzy', 'six']


'six' appears twice.

thanks I overlooked that

Tree.trees, Tree.leaves, Tree.children were once properties, but since these things really could benefit by optionally returning a View with hidden files/directories, we made them methods instead. This also gives some indication that work has to be done to call these, in that it involves a filesystem call. We have gotten rid of .hidden. Also, .trees, .leaves, and .children have been added to Bundle since these are exposed to Treants.

Small changes to make this clearer, even if not all limbs are suitable for Tree objects.

dotsdl · 2017-07-31T01:47:21Z

@orbeckst don't worry...got Python 2.7 working again. Was a small issue with how pickling was being handled in 2 vs. 3. We won't be dropping support at this time.

kain88-de · 2017-08-14T14:16:49Z

@dotsdl and @richardjgowers do you have a list of what is still left todo?

ensures python2/3 compatibility.

…nto spring_cleaning

kain88-de · 2017-10-09T12:37:14Z

src/datreant/core/collections.py

 from .trees import Tree, Leaf
+from .names import treantdir_name


I'm still for using all caps on constants

kain88-de · 2017-10-10T02:45:38Z

src/datreant/core/collections.py


    """
-    _classagglimbs = set()
-    _agglimbs = set()

    def __init__(self, *vegs, **kwargs):


kwargs are unused now. Still needed ?

kain88-de · 2017-10-10T04:04:27Z

src/datreant/core/metadata.py

-    def __init__(self, collection):
-        self._collection = collection
+    @staticmethod
+    def _init_state(jsonfile):


why is this function really needed? I don't see the benefit of it.

I'm also confused here why this function is static. It is only ever called with self as argument.

This method is the init_state method specific to Tags, which gets fed to the JSONFile constructor for use when creating a file from scratch. For Tags this generates an empty list. Categories has its own that generates an empty dict.

kain88-de · 2017-10-10T04:05:11Z

src/datreant/core/state.py

+        super(JSONFile, self).__init__(filename, **kwargs)
+
+        # set _init_state function that given
+        self._init_state = init_state


the object self._init_state is nowhere really called.

self._init_state gets called by Tags and Categories classes, which use JSONFile to talk to their state files. Basically, it's defined at runtime on creation of the JSONFile object.

I'll admit it's a confusing way to go about it. I'm open to alternatives.

yeah, but where is it called? It should also be commented what the intent of this. Otherwise, I'm gonna ask again in the future.

I added a brief docstring to each with this information. Is there more needed, though?

Nope clearer now. A comment here may be nice. But I don't remember exactly why I didn't understand this. Reading the code now it's clear to me.

kain88-de · 2017-10-10T04:10:13Z

src/datreant/core/metadata.py

+        """Return a set giving the Tags in `a` that are not in `b`.
+
+        """
+        from .metadata import AggTags


This line is cool. You are saying import class from this file.

Ha...that's an artifact from before when these classes where in a separate file. I can probably get rid of it.

Added thee upon access of tags or categories since these throw exceptions already upon access. We don't want to do a check for `.datreant` on every access of Treant since this will have a potentially high performance hit for many Treants. Basically, we only raise the issue when it really matters.

…nto spring_cleaning

dotsdl · 2017-11-14T18:59:37Z

Finished updating docs. Next up is to write a simple converter script for old Treants to new form.

kain88-de · 2017-11-14T22:00:06Z

Finished updating docs. Next up is to write a simple converter script for old Treants to new form.

@mimischi there is gonna be a script, I hope I can adjust it to work on a Sim as well.

kain88-de · 2017-11-14T22:06:20Z

docs/treants.rst

-This means that changing the location or name of a Treant can be done at the
-filesystem level. Although this means that one can change the treanttype and
-uuid as well, this is generally not recommended.
+place on disk; they store almost nothing in memory.


Please also point out that we only keep the filehandle alive as shortly as possible. It's good on rare occasions to have this documented.

Good call. Added.

dotsdl · 2017-11-14T22:18:23Z

If we're happy with this, I'm good with merging (finally). Next up will be getting the other PRs that have been waiting in the wings finally in as well.

kain88-de · 2017-11-14T22:19:12Z

sure merge ahead

dotsdl added 4 commits April 12, 2017 23:13

Removed uuids, also now use .datreant directory for Treant

802a22e

More hacking, getting tests to work, removing cruft.

f931bc4

Tests finally pass again. Need to add things I cut to changelog.

cfadd6d

Changed Treant.json to state.json

86c4a69

dotsdl added enhancement fileformat priority high labels Apr 24, 2017

dotsdl added this to the 0.8.0 milestone Apr 24, 2017

dotsdl requested review from kain88-de and richardjgowers April 24, 2017 19:27

dotsdl added 2 commits May 5, 2017 13:36

Now tags and categories have their own state file.

dac55af

There is no state file for the overall Treant anymore. Now, what makes a Treant is the presence of the `.datreant` directory alone. Limbs make whatever files they need inside there to function.

Using __getattribute__ requires __setstate__ gymnastics.

27eef03

All tests pass again, so we shouldn't have any obvious regressions. Now need to add tests that Treants are as hardy as they should be with new machinery.

kain88-de reviewed May 8, 2017

View reviewed changes

kain88-de reviewed May 9, 2017

View reviewed changes

Added treeloc, leafloc path accessors to Tree, View, Bundle.

9b67b2c

These aren't tested yet, so much to do still.

dotsdl and others added 3 commits July 16, 2017 12:55

Bundle, View repr simplifications.

dca3cf6

Tests pass again. Can remove from bundle with names.

ce0f3c6

remove os library usage

2adcc2d

use pathlib directly

kain88-de force-pushed the spring_cleaning branch from 8e1cc05 to 2adcc2d Compare July 21, 2017 16:56

kain88-de added 3 commits July 21, 2017 21:21

use python 3.6 API of os.makedirs

4c2b2be

This will allow to drop the wrapper once the library is py3 only.

fix failing tests

501c7dc

verbose travis pytest output

18a7330

jbarnoud reviewed Jul 21, 2017

View reviewed changes

dotsdl added 2 commits July 29, 2017 15:06

Limbs are generally applicable to Treants and Trees.

d6eac7d

Small changes to make this clearer, even if not all limbs are suitable for Tree objects.

kain88-de force-pushed the spring_cleaning branch from d919b34 to 924cba3 Compare August 21, 2017 12:22

Allow all printable chars in selection parser

a19006f

kain88-de force-pushed the spring_cleaning branch from 924cba3 to a19006f Compare August 21, 2017 13:57

kain88-de and others added 4 commits August 22, 2017 09:41

use range instead of xrange

5891e21

ensures python2/3 compatibility.

add test for __str__ of a treant tag

a153078

Removed all concept of limbs; View and Bundle practically immutable

dff9d9b

Merge branch 'spring_cleaning' of github.com:datreant/datreant.core i…

df30d66

…nto spring_cleaning

kain88-de requested changes Oct 9, 2017

View reviewed changes

kain88-de added 2 commits October 9, 2017 05:48

update gitignore

a0b918f

use capital letter for globals

b016323

kain88-de reviewed Oct 10, 2017

View reviewed changes

kain88-de added 2 commits October 9, 2017 21:12

remove unused kwargs

f421cdb

remove unnecessary import

78f7178

kain88-de approved these changes Oct 10, 2017

View reviewed changes

dotsdl added 3 commits November 14, 2017 10:34

Merge branch 'spring_cleaning' of github.com:datreant/datreant.core i…

e5db06d

…nto spring_cleaning

Updated docs in line with changes in spring cleaning.

890b720

Added docstrings to _init_state staticmethods pointing out their use

ba55c61

kain88-de reviewed Nov 14, 2017

View reviewed changes

Added note on file handles to docs.

5a2d993

dotsdl merged commit 830bef2 into develop Nov 14, 2017

kain88-de deleted the spring_cleaning branch November 21, 2017 22:11

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WIP: Spring cleaning (Treant unification, no uuids, .datreant state) #145

WIP: Spring cleaning (Treant unification, no uuids, .datreant state) #145

dotsdl commented Apr 24, 2017 •

edited by kain88-de

Loading

kain88-de May 8, 2017

kain88-de May 8, 2017

kain88-de left a comment

kain88-de May 9, 2017

dotsdl commented May 28, 2017

codecov-io commented May 28, 2017 •

edited

Loading

kain88-de commented Jun 23, 2017

dotsdl commented Jun 24, 2017

kain88-de commented Jul 21, 2017 •

edited

Loading

kain88-de commented Jul 21, 2017

kain88-de commented Jul 21, 2017

jbarnoud Jul 21, 2017

kain88-de Jul 21, 2017

dotsdl commented Jul 31, 2017

kain88-de commented Aug 14, 2017

kain88-de Oct 9, 2017

kain88-de Oct 10, 2017

kain88-de Oct 10, 2017

kain88-de Oct 10, 2017

dotsdl Nov 14, 2017

kain88-de Oct 10, 2017

dotsdl Nov 14, 2017

kain88-de Nov 14, 2017

dotsdl Nov 14, 2017

kain88-de Nov 14, 2017

kain88-de Oct 10, 2017

dotsdl Nov 14, 2017

dotsdl commented Nov 14, 2017

kain88-de commented Nov 14, 2017

kain88-de Nov 14, 2017

dotsdl Nov 14, 2017

dotsdl commented Nov 14, 2017

kain88-de commented Nov 14, 2017

		@@ -0,0 +1,2 @@
		treantdir_name = '.datreant'
		treantfile_name = 'state.json'

		from .trees import Tree, Leaf
		from .names import treantdir_name

WIP: Spring cleaning (Treant unification, no uuids, .datreant state) #145

WIP: Spring cleaning (Treant unification, no uuids, .datreant state) #145

Conversation

dotsdl commented Apr 24, 2017 • edited by kain88-de Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kain88-de left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dotsdl commented May 28, 2017

codecov-io commented May 28, 2017 • edited Loading

Codecov Report

kain88-de commented Jun 23, 2017

dotsdl commented Jun 24, 2017

kain88-de commented Jul 21, 2017 • edited Loading

kain88-de commented Jul 21, 2017

kain88-de commented Jul 21, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dotsdl commented Jul 31, 2017

kain88-de commented Aug 14, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dotsdl commented Nov 14, 2017

kain88-de commented Nov 14, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dotsdl commented Nov 14, 2017

kain88-de commented Nov 14, 2017

dotsdl commented Apr 24, 2017 •

edited by kain88-de

Loading

codecov-io commented May 28, 2017 •

edited

Loading

kain88-de commented Jul 21, 2017 •

edited

Loading