Skip to content

Commit

Permalink
Remove Group object entirely (#130)
Browse files Browse the repository at this point in the history
* Removed Group.

* Removed Groups entirely from docs.

* PEP8 fixes to tests
  • Loading branch information
dotsdl authored and kain88-de committed Feb 22, 2017
1 parent 819b1b3 commit a09fb22
Show file tree
Hide file tree
Showing 13 changed files with 98 additions and 749 deletions.
21 changes: 0 additions & 21 deletions docs/api_treants.rst
Original file line number Diff line number Diff line change
Expand Up @@ -38,24 +38,3 @@ Treants to access their categories.
.. autoclass:: datreant.core.limbs.Categories
:members:
:inherited-members:

.. _Group_api:

Group
-----
The class :class:`datreant.core.Group` is a Treant with the ability to store
member locations as a persistent Bundle within its state file.

.. autoclass:: datreant.core.Group
:members:
:inherited-members:

Members
```````
The class :class:`datreant.core.limbs.MemberBundle` is the interface used by a
Group to manage its members. Its API matches that of
:class:`datreant.core.Bundle`.

.. autoclass:: datreant.core.limbs.MemberBundle
:members:
:inherited-members:
22 changes: 4 additions & 18 deletions docs/faq.rst
Original file line number Diff line number Diff line change
Expand Up @@ -20,11 +20,6 @@ Frequently Asked Questions
whole database somehow, or slurp the pieces of data out that you want.
Most database solutions can be rather slow to do this.

*Treants are independent.*
Although Groups are aware of their members, Treants work independently
from one another. If you want to use only basic Treants, that works
just fine. If you want to use Groups, that works, too.

*Treants have a structure in the filesystem.*
This means that all the shell tools we know and love are available to
work with their contents, which might include plaintext files, figures,
Expand All @@ -33,21 +28,12 @@ Frequently Asked Questions
filesystem is, at least when it comes to storage.

2. What are some disadvantages of datreant's design?

*Treants could be anywhere in the filesystem.*
This is mostly a problem for Groups, which allow aggregation of other
Treants. If a member is moved, the Group has no way of knowing where it
went; we've built machinery to help it find its members, but these will
always be limited to filesystem search methods (some quite good, but
still). If these objects lived in a single database, this wouldn't be
an issue.

*Queries on object metadata will be slower than a central database.*
We want Groups and Bundles (in-memory Groups, basically) to be able to
run queries against their members' characteristics, returning subsets
matching the query. Since these queries have to be applied against
these objects and not against a single table somewhere, it will be
relatively slow.
We Bundles to be able to run queries against their members'
characteristics, returning subsets matching the query. Since these
queries have to be applied against these objects and not against a
single table somewhere, it will be relatively slow.

*File locking is less efficient for multiple-read/write under load than a smart daemon process/scheduler.*
The assumption we make is that Treants are primarily read, and only
Expand Down
82 changes: 0 additions & 82 deletions docs/groups.rst

This file was deleted.

1 change: 0 additions & 1 deletion docs/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -76,7 +76,6 @@ for more information.
trees
views
bundles
groups
api

.. toctree::
Expand Down
5 changes: 2 additions & 3 deletions src/datreant/core/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,6 @@
===================================================================
.. SeeAlso:: :class:`datreant.core.treants.Treant`
:class:`datreant.core.treants.Group`
"""
# global registries of classes. Must be imported first! Other modules will try
Expand All @@ -22,11 +21,11 @@

# Bring some often used objects into the current namespace
from .manipulators import discover
from .treants import Treant, Group
from .treants import Treant
from .trees import Veg, Leaf, Tree
from .collections import View, Bundle
from . import attach

__all__ = ['Treant', 'Group', 'Tree', 'Leaf', 'Bundle', 'discover', 'Veg',
__all__ = ['Treant', 'Tree', 'Leaf', 'Bundle', 'discover', 'Veg',
'attach', 'View']
__version__ = "0.8.0-dev" # NOTE: keep in sync with RELEASE in setup.py
7 changes: 2 additions & 5 deletions src/datreant/core/attach.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,18 +3,15 @@
"""

from .treants import Treant, Group
from .treants import Treant
from .collections import Bundle
from .limbs import Tags, Categories, MemberBundle
from .limbs import Tags, Categories
from .agglimbs import AggTags, AggCategories

# Treants get tags and categories
for limb in (Tags, Categories):
Treant._attach_limb_class(limb)

# Groups get members
Group._attach_limb_class(MemberBundle)

# Bundles get tags and categories aggregation limbs
for agglimb in (AggTags, AggCategories):
Bundle._attach_agglimb_class(agglimb)
36 changes: 1 addition & 35 deletions src/datreant/core/collections.py
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
"""
The Bundle object is the primary manipulator for Treants in aggregate. They are
returned as queries to Groups and other Bundles. They offer convenience methods
returned as queries to Bundles. They offer convenience methods
for dealing with many Treants at once. Views give the same kind of aggregation
conveniences for Trees and Leaves.
Expand Down Expand Up @@ -1060,40 +1060,6 @@ def searchtime(self, value):
else:
raise TypeError("Must give a number or `None` for searchtime")

def flatten(self, exclude=None):
"""Return a flattened version of this Bundle.
The resulting Bundle will have all members of any member Groups,
without the Groups.
Parameters
----------
exclude : list
uuids of Groups to leave out of flattening; these will not in the
resulting Bundle.
Returns
-------
flattened : Bundle
the flattened Bundle with no Groups
"""
if not exclude:
exclude = list()

guuids = list(exclude)
memberlist = self._list()
flattened = Bundle(limbs=self.limbs)

for member in memberlist:
if hasattr(member, 'members') and member.uuid not in exclude:
guuids.append(member.uuid)
flattened += member.members.flatten(guuids)
elif not hasattr(member, 'members'):
flattened.add(member)

return flattened

@property
def view(self):
"""Obtain a View giving the Tree for each Treant in this Bundle.
Expand Down
83 changes: 2 additions & 81 deletions src/datreant/core/filesystem.py
Original file line number Diff line number Diff line change
Expand Up @@ -96,7 +96,7 @@ class Foxhound(object):
This object is used by Treants to find Treants, even when they are no
longer in their last known location.
Groups and Bundles uses this class to find members that have moved. All
Bundles uses this class to find members that have moved. All
TreantFiles use this class to find their file on disk when it moves.
"""
Expand Down Expand Up @@ -144,13 +144,9 @@ def fetch(self, as_treants=True):
instead of paths for *as_treants* == True.
"""
from .limbs import MemberBundle
from .collections import Bundle

if isinstance(self.caller, MemberBundle):
results = self._find_Group_members()
elif isinstance(self.caller, Bundle):
results = self._find_Bundle_members()
results = self._find_Bundle_members()

if as_treants:
conts = path2treant(*results.values())
Expand Down Expand Up @@ -220,81 +216,6 @@ def _find_TreantFile(self):
"""
pass

def _find_Group_members(self):
"""Find Treants that are members of a Group.
For finding Group members, the Foxhound begins by looking for
Treants among the paths it was given. Treants that can't be found
are then searched for starting downward from the Group's location, with
subsequent downward searches proceeding from the parent directory.
This process continues until either all members are found, the
filesystem is exhaustively searched, or the Foxhound times out.
:Returns:
*outpaths*
dictionary giving Treant uuids as keys and absolute paths to
their state files as values; ``None`` as a value indicates
that no state file could be found.
"""
# search last-known locations
outpaths = self._check_paths()

# get current time
currtime = time.time()

# walk downwards on an upward path through filesystem from the Group's
# basedir
uuids = [str(x) for x in outpaths if not outpaths[x]]
path = self.caller._treant.location
prev = None
timedout = False
while prev != path and uuids and not timedout:

top = True
for root, dirs, files in scandir.walk(path):
# if search runs over timeout, call it off
if ((time.time() - currtime) > self.timeout and
self.timeout is not None):

self.caller._logger.info(
"Search for missing members timed" +
" out at {}".format(self.timeout) +
" seconds.")
timedout = True
break

# if we've found everything, finish
if not uuids:
break

found = []
# no need to visit already-visited tree
if top and prev:
dirs.remove(os.path.basename(prev))
top = False

for uuid in uuids:
candidate = [os.path.join(root, x) for x in files
if ((uuid in x) and
('.json' in x) and
(x[0] != '.'))]

if candidate:
outpaths[uuid] = os.path.abspath(candidate[0])
found.append(uuid)

for item in found:
uuids.remove(item)

prev = path
path = os.path.split(path)[0]

# TODO: post-check? Since Groups know the treanttypes of their
# members, should we compare these to what is in outpaths?

return outpaths

def _find_Bundle_members(self):
"""Find Treants that are members of a Bundle.
Expand Down

0 comments on commit a09fb22

Please sign in to comment.