Skip to content
Browse files

fork docs

  • Loading branch information...
1 parent e9fd618 commit 63e86963d11c1a41ed60617aba41559be15abd1c @mcdonc mcdonc committed Jun 20, 2012
Showing with 139 additions and 1,062 deletions.
  1. +1 −0 docs/.gitignore
  2. +34 −32 docs/api.rst
  3. +2 −2 docs/changes.rst
  4. +0 −40 docs/code/index_attributes.py
  5. +0 −46 docs/code/index_callbacks.py
  6. +35 −18 docs/conf.py
  7. +31 −28 docs/genealogy.rst
  8. +7 −10 docs/glossary.rst
  9. +14 −16 docs/index.rst
  10. +15 −17 docs/install.rst
  11. +0 −157 docs/overview.rst
  12. +0 −61 docs/upgrade.rst
  13. +0 −635 docs/usage.rst
View
1 docs/.gitignore
@@ -1 +1,2 @@
_build/
+_themes/
View
66 docs/api.rst
@@ -1,9 +1,9 @@
.. _api_catalog_section:
-:mod:`repoze.catalog.catalog`
------------------------------
+:mod:`hypatia.catalog`
+----------------------
-.. automodule:: repoze.catalog.catalog
+.. automodule:: hypatia.catalog
.. autoclass:: Catalog
:members:
@@ -18,11 +18,23 @@
Retrieve an index or return failobj.
+ .. automethod:: clear_indexes
-:mod:`repoze.catalog.query`
----------------------------
+ .. automethod:: index_doc
-.. module:: repoze.catalog.query
+ .. automethod:: unindex_doc
+
+ .. automethod:: reindex_doc
+
+ .. autoclass:: CatalogQuery
+ :members:
+
+ .. automethod:: __call__
+
+:mod:`hypatia.query`
+--------------------
+
+.. module:: hypatia.query
Comparators
~~~~~~~~~~~
@@ -75,59 +87,49 @@ Other Helpers
.. _api_fieldindex_section:
-:mod:`repoze.catalog.indexes.field`
------------------------------------
+:mod:`hypatia.field`
+--------------------
-.. automodule:: repoze.catalog.indexes.field
+.. automodule:: hypatia.field
- .. autoclass:: CatalogFieldIndex
+ .. autoclass:: FieldIndex
:members:
.. _api_keywordindex_section:
-:mod:`repoze.catalog.indexes.keyword`
+:mod:`hypatia.keyword`
-------------------------------------
-.. automodule:: repoze.catalog.indexes.keyword
+.. automodule:: hypatia.keyword
- .. autoclass:: CatalogKeywordIndex
+ .. autoclass:: KeywordIndex
:members:
.. _api_textindex_section:
-:mod:`repoze.catalog.indexes.text`
+:mod:`hypatia.text`
-----------------------------------
-.. automodule:: repoze.catalog.indexes.text
+.. automodule:: hypatia.text
- .. autoclass:: CatalogTextIndex
+ .. autoclass:: TextIndex
:members:
.. _api_facetindex_section:
-:mod:`repoze.catalog.indexes.facet`
+:mod:`hypatia.facet`
-------------------------------------
-.. automodule:: repoze.catalog.indexes.facet
+.. automodule:: hypatia.facet
- .. autoclass:: CatalogFacetIndex
+ .. autoclass:: FacetIndex
:members:
-:mod:`repoze.catalog.indexes.path`
+:mod:`hypatia.path`
----------------------------------
-.. automodule:: repoze.catalog.indexes.path
+.. automodule:: hypatia.path
- .. autoclass:: CatalogPathIndex
+ .. autoclass:: PathIndex
:members:
-.. _api_document_section:
-
-:mod:`repoze.catalog.document`
-------------------------------
-
-.. automodule:: repoze.catalog.document
-
- .. autoclass:: DocumentMap
- :members:
-
View
4 docs/changes.rst
@@ -1,4 +1,4 @@
-:mod:`repoze.catalog` Change History
-====================================
+:mod:`hypatia` Change History
+=============================
.. literalinclude:: ../CHANGES.txt
View
40 docs/code/index_attributes.py
@@ -1,40 +0,0 @@
-from repoze.catalog.catalog import FileStorageCatalogFactory
-from repoze.catalog.catalog import ConnectionManager
-
-from repoze.catalog.indexes.field import CatalogFieldIndex
-from repoze.catalog.indexes.text import CatalogTextIndex
-
-factory = FileStorageCatalogFactory('catalog.db', 'mycatalog')
-
-_initialized = False
-
-def initialize_catalog():
- global _initialized
- if not _initialized:
- # create a catalog
- manager = ConnectionManager()
- catalog = factory(manager)
- # set up indexes
- catalog['flavors'] = CatalogFieldIndex('flavor')
- catalog['texts'] = CatalogTextIndex('text')
- # commit the indexes
- manager.commit()
- manager.close()
- _initialized = True
-
-class Content(object):
- def __init__(self, flavor, text):
- self.flavor = flavor
- self.text = text
-
-if __name__ == '__main__':
- initialize_catalog()
- manager = ConnectionManager()
- catalog = factory(manager)
- content = {
- 1:Content('peach', 'i am so very very peachy'),
- 2:Content('pistachio', 'i am nutty'),
- }
- for docid, doc in content.items():
- catalog.index_doc(docid, doc)
- manager.commit()
View
46 docs/code/index_callbacks.py
@@ -1,46 +0,0 @@
-from repoze.catalog.catalog import FileStorageCatalogFactory
-from repoze.catalog.catalog import ConnectionManager
-
-from repoze.catalog.indexes.field import CatalogFieldIndex
-from repoze.catalog.indexes.text import CatalogTextIndex
-
-factory = FileStorageCatalogFactory('catalog.db', 'mycatalog')
-
-class Content(object):
- def __init__(self, flavor, text):
- self.theflavor = flavor
- self.thetext = text
-
-def get_flavor(object, default):
- return getattr(object, 'theflavor', default)
-
-def get_text(object, default):
- return getattr(object, 'thetext', default)
-
-_initialized = False
-
-def initialize_catalog():
- global _initialized
- if not _initialized:
- # create a catalog
- manager = ConnectionManager()
- catalog = factory(manager)
- # set up indexes
- catalog['flavors'] = CatalogFieldIndex(get_flavor)
- catalog['text'] = CatalogTextIndex(get_text)
- # commit the indexes
- manager.commit()
- manager.close()
- _initialized = True
-
-if __name__ == '__main__':
- initialize_catalog()
- manager = ConnectionManager()
- catalog = factory(manager)
- content = {
- 1:Content('peach', 'i am so very very peachy'),
- 2:Content('pistachio', 'i am nutty'),
- }
- for docid, doc in content.items():
- catalog.index_doc(docid, doc)
- manager.commit()
View
53 docs/conf.py
@@ -1,6 +1,6 @@
# -*- coding: utf-8 -*-
#
-# repoze.catalog documentation build configuration file
+# hypatia documentation build configuration file
#
# This file is execfile()d with the current directory set to its containing dir.
#
@@ -16,18 +16,25 @@
# is relative to the documentation root, use os.path.abspath to make it
# absolute, like shown here.
+# Add and use Pylons theme
if 'sphinx-build' in ' '.join(sys.argv): # protect against dumb importers
- parent = os.path.dirname(os.path.dirname(__file__))
- sys.path.append(os.path.abspath(parent))
- wd = os.getcwd()
- os.chdir(parent)
- os.system('%s setup.py test -q' % sys.executable)
- os.chdir(wd)
+ from subprocess import call, Popen, PIPE
- for item in os.listdir(parent):
- if item.endswith('.egg'):
- sys.path.append(os.path.join(parent, item))
+ p = Popen('which git', shell=True, stdout=PIPE)
+ git = p.stdout.read().strip()
+ cwd = os.getcwd()
+ _themes = os.path.join(cwd, '_themes')
+ if not os.path.isdir(_themes):
+ call([git, 'clone', 'git://github.com/Pylons/pylons_sphinx_theme.git',
+ '_themes'])
+ else:
+ os.chdir(_themes)
+ call([git, 'checkout', 'master'])
+ call([git, 'pull'])
+ os.chdir(cwd)
+
+ sys.path.append(os.path.abspath('_themes'))
# General configuration
# ---------------------
@@ -46,14 +53,14 @@
master_doc = 'index'
# General substitutions.
-project = 'repoze.catalog'
-copyright = '2008-2012, Agendaless Consulting'
+project = 'hypatia'
+copyright = '2008-2012, Zope Foundation and Contributors'
# The default replacements for |version| and |release|, also used in various
# other places throughout the built documents.
#
# The short X.Y version.
-version = '0.8.2'
+version = '0.1a1'
# The full version, including alpha/beta/rc tags.
release = version
@@ -70,6 +77,8 @@
# for source files.
#exclude_dirs = []
+exclude_patterns = ['_themes/README.rst',]
+
# The reST default role (used for this markup: `text`) to use for all documents.
#default_role = None
@@ -91,10 +100,18 @@
# Options for HTML output
# -----------------------
+# Add and use Pylons theme
+html_theme_path = ['_themes']
+html_theme = 'pylons'
+html_theme_options = dict(
+ github_url='https://github.com/Pylons/hypatia',
+# in_progress='true'
+ )
+
# The style sheet to use for HTML and HTML Help pages. A file of that name
# must exist either in Sphinx' static/ path, or in one of the custom paths
# given in html_static_path.
-html_style = 'repoze.css'
+#html_style = 'repoze.css'
# The name for this set of Sphinx documents. If None, it defaults to
# "<project> v<release> documentation".
@@ -105,7 +122,7 @@
# The name of an image file (within the static path) to place at the top of
# the sidebar.
-html_logo = '.static/logo_hi.gif'
+#html_logo = '.static/logo_hi.gif'
# The name of an image file (within the static path) to use as favicon of the
# docs. This file should be a Windows icon file (.ico) being 16x16 or 32x32
@@ -115,7 +132,7 @@
# Add any paths that contain custom static files (such as style sheets) here,
# relative to this directory. They are copied after the builtin static files,
# so a file named "default.css" will overwrite the builtin "default.css".
-html_static_path = ['.static']
+#html_static_path = ['.static']
# If not '', a 'Last updated on:' timestamp is inserted at every page bottom,
# using the given strftime format.
@@ -153,7 +170,7 @@
#html_file_suffix = ''
# Output file base name for HTML help builder.
-htmlhelp_basename = 'repozecatalogdoc'
+htmlhelp_basename = 'hypatiadoc'
# Options for LaTeX output
@@ -168,7 +185,7 @@
# Grouping the document tree into LaTeX files. List of tuples
# (source start file, target name, title, author, document class [howto/manual]).
latex_documents = [
- ('index', 'repozecatalog.tex', 'repoze.catalog Documentation', 'Agendaless Consulting', 'manual'),
+ ('index', 'hypatia.tex', 'hypatia Documentation', 'Agendaless Consulting', 'manual'),
]
# The name of an image file (relative to this directory) to place at the top of
View
59 docs/genealogy.rst
@@ -1,32 +1,35 @@
-Genealogy of :mod:`repoze.catalog`
-==================================
+Genealogy of :mod:`hypatia`
+===========================
-All versions of Zope depend heavily on the Zope Object Database (ZODB).
-Because ZODB is less a database and more a persistent object store (it
-doesn't possess a query language; Python *is* its query language), it
-has been necessary to create indexing and searching facilities for data
-stored in ZODB.
+Hypatia is derived from :term:`zope.index` and :term:`repoze.catalog`.
-The first iteration of searching and indexing for ZODB-based
-applications (at least post-Principia, which had something named
-Tabula, which I never actually used) was the ZCatalog. The ZCatalog
-was entirely tied to Zope2, and still remains in heavy use today
-within Zope 2 applications such as Plone.
+Hypatia depends heavily on the Zope Object Database (ZODB). Because ZODB is
+less a database and more a persistent object store (it doesn't possess a
+query language; Python *is* its query language), it has been necessary to
+create indexing and searching facilities for data stored in ZODB.
-The second iteration was ``zope.app.catalog``, which was a ZCatalog
-do-over for Zope 3 applications.
+The first iteration of searching and indexing for ZODB-based applications (at
+least post-Principia, which had something named Tabula, which I never
+actually used) was the ZCatalog. The ZCatalog was entirely tied to Zope2,
+and still remains in heavy use today within Zope 2 applications such as
+Plone.
-Neither of these searching and indexing packages are particularly easy
-to use outside of a Zope application. Each makes various assumptions
-about the content objects that need indexing or the environment that
-aren't appropriate for arbitrary applications. For instance, ZCatalog
-wants objects you want to catalog to have a ``getPhysicalPath`` method
-which returns a "path". An instance of ``zope.app.catalog`` makes the
-assumption that that it's located within a Zope 3 "site" object within
-a ZODB, and assumes that you want query result sets to be sets of
-Python references to the original object you indexed. In other words,
-these packages assume too much to be maximally useful outside the
-context in which they were developed. `Repoze <http://repoze.org>`_ is
-a project which has as a stated goal making it easier for non-Zope
-Python developers to use Zope technologies outside Zope, so this
-seemed like a natural thing to do under the Repoze flag.
+The second iteration was ``zope.app.catalog``, which was a ZCatalog do-over
+for Zope 3 applications.
+
+Neither of these searching and indexing packages are particularly easy to use
+outside of a Zope application. Each makes various assumptions about the
+content objects that need indexing or the environment that aren't appropriate
+for arbitrary applications. For instance, ZCatalog wants objects you want to
+catalog to have a ``getPhysicalPath`` method which returns a "path". An
+instance of ``zope.app.catalog`` makes the assumption that that it's located
+within a Zope 3 "site" object within a ZODB, and assumes that you want query
+result sets to be sets of Python references to the original object you
+indexed. In other words, these packages assume too much to be maximally
+useful outside the context in which they were developed. `Repoze
+<http://repoze.org>`_ is a project which has as a stated goal making it
+easier for non-Zope Python developers to use Zope technologies outside Zope,
+so this seemed like a natural thing to do under the Repoze flag.
+
+Hypatia is a reboot of both zope.index and repoze.catalog with no backwards
+compatibility constraints.
View
17 docs/glossary.rst
@@ -44,15 +44,12 @@ Glossary
documents can be searched for by their position in the graph.
zope.index
- The `underlying indexing machinery
- <http://pypi.python.org/pypi/zope.index>`_ that
- :mod:`repoze.catalog` uses.
+ One `package
+ <http://pypi.python.org/pypi/zope.index>`_ that hypatia was forked from.
- zope.app.catalog
- The `cataloging implementation
- <http://pypi.python.org/pypi/zope.app.catalog>`_ on which
- :mod:`repoze.catalog` is based (although it doesn't use any of
- its code).
+ repoze.catalog
+ `Another package
+ <http://pypi.python.org/pypi/repoze.catalog>`_ that hypatia was forked from.
Virtualenv
An isolated Python environment. Allows you to control which
@@ -65,6 +62,6 @@ Glossary
expression which is used to generate a query object.
Query Object
- An object used as an argument to the :meth:`repoze.catalog.Catalog.query`
- method's ``queryobject`` parameter.
+ An object used as an argument to the
+ :meth:`hypatia.CatalogQuery.__call__` method's ``queryobject`` parameter.
View
30 docs/index.rst
@@ -1,35 +1,32 @@
.. _index:
-==============
-repoze.catalog
-==============
+=======
+Hypatia
+=======
-:mod:`repoze.catalog` is a Python indexing and searching framework.
-It relies on :term:`zope.index` and most of its internals are taken
-from :term:`zope.app.catalog`. Unlike ``zope.app.catalog``, however,
-it is meant to be useful outside of the larger Zope framework within
-arbitrary Python applications.
+:mod:`hypatia` is a Python indexing and searching framework.
Narrative documentation
-----------------------
-Narrative documentation explaining how to use :mod:`repoze.catalog`.
+Narrative documentation explaining how to use :mod:`hypatia`.
+
+.. warning::
+
+ This is under construction during the alpha phase of Hypatia.
.. toctree::
:maxdepth: 2
- overview
- install
- upgrade
- usage
genealogy
+ install
glossary
changes
API documentation
-----------------
-API documentation for :mod:`repoze.catalog`.
+API documentation for :mod:`hypatia`.
.. toctree::
:maxdepth: 2
@@ -39,9 +36,10 @@ API documentation for :mod:`repoze.catalog`.
Source Code and Issue Tracking
------------------------------
-Source code is available from https://github.com/repoze/repoze.catalog
+Source code is available from https://github.com/Pylons/hypatia
+
+File bugs via https://github.com/Pylons/hypatia/issues
-File bugs via https://github.com/repoze/repoze.catalog/issues
Indices and tables
==================
View
32 docs/install.rst
@@ -1,36 +1,34 @@
-Installing :mod:`repoze.catalog`
-================================
+Installing :mod:`hypatia`
+=========================
How To Install
--------------
-You will need `Python <http://python.org>`_ version 2.4 or better to
-run :mod:`repoze.catalog`. Development of :mod:`repoze.catalog` is
-done primarily under Python 2.6, so that version is recommended.
-:mod:`repoze.catalog` also runs under Python 2.4 and 2.5 with limited
-functionality. It does *not* run under Python 3.X.
+You will need `Python <http://python.org>`_ version 2.6 or better to run
+:mod:`hypatia`. Development of :mod:`hypatia` is done
+primarily under Python 2.7, so that version is recommended. It does *not*
+run under Python 3.X.
-.. warning:: To succesfully install :mod:`repoze.catalog`, you will need an
+.. warning:: To succesfully install :mod:`hypatia`, you will need an
environment capable of compiling Python C code. See the
documentation about installing, e.g. ``gcc`` and ``python-devel``
for your system. You will also need :term:`setuptools` installed
within your Python system in order to run the ``easy_install``
command.
-It is advisable to install :mod:`repoze.catalog` into a
-:term:`virtualenv` in order to obtain isolation from any "system"
-packages you've got installed in your Python version (and likewise, to
-prevent :mod:`repoze.catalog` from globally installing versions of
-packages that are not compatible with your system Python).
+It is advisable to install :mod:`hypatia` into a :term:`virtualenv` in order
+to obtain isolation from any "system" packages you've got installed in your
+Python version (and likewise, to prevent :mod:`hypatia` from globally
+installing versions of packages that are not compatible with your system
+Python).
After you've got the requisite dependencies installed, you may install
-:mod:`repoze.catalog` into your Python environment using the following
+:mod:`hypatia` into your Python environment using the following
command::
- $ easy_install repoze.catalog
+ $ easy_install hypatia
What Gets Installed
-------------------
-When you ``easy_install`` :mod:`repoze.catalog`, various Zope
-libraries and ZODB are installed.
+When you ``easy_install`` :mod:`hypatia` and ZODB are installed.
View
157 docs/overview.rst
@@ -1,157 +0,0 @@
-A Tour of :mod:`repoze.catalog`
-===============================
-
-:mod:`repoze.catalog` borrows heavily from ``zope.app.catalog`` and
-depends wholly on the ``zope.index`` package for its index
-implementations, but assumes less about how you want your indexing and
-querying to behave. In this spirit, you can index any Python object;
-it needn't implement any particular interface except perhaps one you
-define yourself conventionally. :mod:`repoze.catalog` does less than
-any of its predecessors, in order to make it more useful for arbitrary
-Python applications. Its implemented in terms of ZODB objects, and
-the ZODB will store the derived index data, but it assumes little
-else. You should be able to use it in any Python application. The
-fact that it uses ZODB is ancillary: it's akin to Xapian using "flint"
-or "quartz" backends.
-
-Indexing
---------
-
-To perform indexing of objects, you set up a catalog with some number
-of indexes, each of which is capable of calling a callback function to
-obtain data about an object being cataloged::
-
- from repoze.catalog.indexes.field import CatalogFieldIndex
- from repoze.catalog.indexes.text import CatalogTextIndex
- from repoze.catalog.catalog import Catalog
-
- def get_flavor(object, default):
- return getattr(object, 'flavor', default)
-
- def get_description(object, default):
- return getattr(object, 'description', default)
-
- catalog = Catalog()
- catalog['flavors'] = CatalogFieldIndex(get_flavor)
- catalog['description'] = CatalogTextIndex(get_description)
-
-Note that ``get_flavor`` and ``get_description`` will be called for each
-object you attempt to index. Each of them attempts to grab an
-attribute from the object being indexed, and returns a default if no
-such attribute exists.
-
-Once you've got a catalog set up, you can begin to index Python
-objects (aka "documents")::
-
- class IceCream(object):
- def __init__(self, flavor, description):
- self.flavor = flavor
- self.description = description
-
- peach = IceCream('peach', 'This ice cream has a peachy flavor')
- catalog.index_doc(1, peach)
-
- pistachio = IceCream('pistachio', 'This ice cream tastes like pistachio nuts')
- catalog.index_doc(2, pistachio)
-
-Note that when you call ``index_doc``, you pass in a ``docid`` as the
-first argument, and the object you want to index as the second
-argument. When we index the ``peach`` object above we index it with
-the docid ``1``. Each docid must be unique within a catalog; when you
-query a :mod:`repoze.catalog` catalog, you'll get back a sequence of
-document ids that match the query you supplied, which you'll
-presumably need to map back to the content object in order to make
-sense of the response; you're responsible for keeping track of which
-objects map to which document id yourself.
-
-Querying
---------
-
-Once you've got some number of documents indexed, you can perform queries
-against an existing catalog. A query is performed by passing a query argument
-and optional keyword arguments to the ``query`` method of the catalog object::
-
- from repoze.catalog.query import Eq
- catalog.query(Eq('flavor', 'peach'))
-
-The argument passed to ``query`` above is a :term:`query object`.
-This particular query object is a :class:`repoze.catalog.query.Eq`
-object, which is a *comparator* meaning "equals". The first argument
-to the ``Eq`` object is an index name, the second argument is a value.
-In english, this query represents "a document indexed in the
-``flavor`` index with the value ``peach``". Other arguments to
-:meth:`repoze.catalog.Catalog.query` may be special values that
-specify sort ordering and query limiting.
-
-In the above example, we specified no particular sort ordering or
-limit, and we're essentially asking the catalog to return us all the
-documents that match the word ``peach`` as a field within the field
-index named ``flavor``. Other types of indexes can be queried
-similarly::
-
- from repoze.catalog.query import Contains
- catalog.query(Contains('description', 'nuts'))
-
-The result of calling the ``query`` method is a two tuple. The first
-element of the tuple is the number of document ids in the catalog
-which match the query. The second element is an iterable: each
-iteration over this iterable returns a document id. The results of
-``catalog.query(Contains('description', 'nuts'))`` might return::
-
- (1, [2])
-
-The first element in the tuple indicates that there is one document in
-the catalog that matches the description 'nuts'. The second element
-in the tuple (here represented as a list, although it's more typically
-a generator) is a sequence of document ids that match the query.
-
-You can combine search parameters to further limit a query::
-
- from repoze.catalog.query import Contains, Eq, Intersection
- catalog.query(Eq('flavor', 'peach') & Contains('description', 'nuts'))
-
-This would return a result representing all the documents indexed
-within the catalog with the flavor of peach and a description of nuts.
-
-Index Types
------------
-
-Out of the box, ``repoze.catalog`` supports five index types: field indexes,
-keyword indexes, text indexes, facet indexes, and path indexes. Field indexes
-are meant to index single discrete values. Keys are stored in order, allowing
-for the full suite of range and comparison operators to be used. Keyword
-indexes index sequences of values which can be queried for any of the values
-in each sequence indexed. Text indexes index text using the
-``zope.index.text`` index type, and can be queried with arbitrary textual
-terms. Text indexes can use various splitting and normalizing strategies to
-collapse indexed texts for better querying. Facet indexes are much like
-keyword indexes, but also allow for "faceted" indexing and searching, useful
-for performing narrowing searches when there is a well-known set of allowable
-values (the "facets"). Path indexes allow you to index documents as part of a
-graph, and return documents that are contained in a portion of the graph.
-
-.. note:: The existing facet index implementation narrowing support is
- naive. It is not meant to be used in catalogs that must use it to
- get count information for over, say, 30K documents, for performance
- reasons.
-
-Helper Facilities
------------------
-
-:mod:`repoze.catalog` provides some helper facilities which help you
-integrate a catalog into an arbitrary Python application. The most
-obvious is a ``FileStorageCatalogFactory``, which makes it reasonably
-easy to create a Catalog object within an arbitrary Python
-application. Using this facility, you don't have to know anything
-about ZODB to use :mod:`repoze.catalog`. If you have an existing ZODB
-application, however, you can ignore this facility entirely and use
-the Catalog implementation directly.
-
-:mod:`repoze.catalog` provides a ``DocumentMap`` object which can be
-used to map document ids to "addresses". An address is any value that
-can be used to resolve the document id back into to a Python object.
-In Zope, an address is typically a traversal path. This facility
-exists in :mod:`repoze.catalog.document.DocumentMap`.
-
-
-
View
61 docs/upgrade.rst
@@ -1,61 +0,0 @@
-Upgrading From Earlier Versions
-===============================
-
-Catalogs generated with :mod:`repoze.catalog` versions prior to 0.8.0 require
-migration in order to add some house keeping data to the indexes that was not
-needed prior to 0.8.0. The easiest way to migrate a catalog is actually to
-delete the catalog and reindex your content objects. This is sufficient in
-the vast majority of cases, since the catalog merely indexes data which itself
-lives outside of the catalog, either in the same ZODB instance or in another
-database altogether.
-
-For particularly large datasets, rebuilding your catalog from scratch can
-potentially take a long time. In order to make this less painful,
-:mod:`repoze.catalog` does contain a migration utility which can migrate an
-existing catalog in a much shorter period of time. This utility comes in three
-different flavors which can be used according to your use case. The primary
-problem being solved by the migration utility is making the indexes aware of
-all of the docids stored in the catalog, not just just the docids of documents
-which have values in a particular index. To this end, the three different
-flavors of migration utility represent three different ways to obtain the set
-of all document ids stored in the catalog.
-
-
-Migrate Using a Document Map
-----------------------------
-
-It will be the easiest and fastest to migrate your pre-0.8.0 catalog if you
-are using a document map which is an instance of
-``repoze.bfg.document.DocumentMap``::
-
- from repoze.catalog.migration import migrate_to_0_8_0_from_document_map
-
- migrate_to_0_8_0_from_document_map(catalog, document_map)
-
-
-Migrate Using Self-Generated Set of Docids
-------------------------------------------
-
-Alternatively, if you do not use a document map, you can use a self-generated
-set of document ids. It is up to your application to calculate the set of all
-document ids used in the catalog::
-
- from repoze.catalog.migration import migrate_to_0_8_0_from_docids
-
- migrate_to_0_8_0_from_docids(catalog, docids)
-
-
-Let the Migration Utility Calculate the Set of Docids
------------------------------------------------------
-
-Your last option is to allow the migration utility to calculate the set of all
-docids by examinging the indexes in the catalog. If you can assume that every
-document indexed by the catalog has a value in at least one index, then this
-method, while potentially slow, should succeed in calculating the superset of
-all document ids in the catalog::
-
- from repoze.catalog.migration import migrate_to_0_8_0
-
- migrate_to_0_8_0(catalog)
-
-
View
635 docs/usage.rst
@@ -1,635 +0,0 @@
-.. _usage:
-
-Using :mod:`repoze.catalog`
-===========================
-
-:mod:`repoze.catalog` is an indexing and search system for Python. It
-is inspired by (and uses much code from) Zope's
-:term:`zope.app.catalog`, and uses other :term:`Zope` libraries to do
-much of its work. It manages its own persistence: it stores catalog
-information into a :term:`ZODB` database.
-
-In order to make use of :mod:`repoze.catalog`, your application will
-be required to create objects that are willing to be indexed, and it
-will be responsible for providing each of these objects a unique
-integer identifier, and maintaining the association between the object
-and the unique identifier for the lifetime of your application.
-Objects which are willing to be indexed must either have a particular
-attribute which is guaranteed to have a value *or* you must provide a
-callback that is willing to inspect the content for a value.
-
-The result of searching a catalog is a sequence of integers that
-represent all the document ids that match the query. Your application
-is responsible for being able to (re-) resolve these integers into
-content objects.
-
-Indexing
---------
-
-Here's a simple example of indexing data within your application.
-This example sets up two indexes.
-
-The first index for ``flavor`` is a :term:`field index`. The second
-index, ``text``, is a :term:`text index`.
-
-.. literalinclude:: code/index_attributes.py
- :linenos:
- :language: python
-
-Here's a more complicated example. It uses callbacks to adapt
-cataloged objects to values rather than directly inspecting attributes
-of the content object. We use the same types of indexes as the
-previous example, but we set up callbacks that allow us to adapt
-content to a result instead of examining the object for an attribute
-directly. This is useful in the case that your content objects don't
-have attributes that match exactly what you want to index:
-
-.. literalinclude:: code/index_callbacks.py
- :linenos:
- :language: python
-
-Searching
----------
-
-Searching for values from a previously indexed corpus of content is
-significantly easier than indexing. There are a number of ways to
-perform searches.
-
-Search Using the :meth:`repoze.catalog.Catalog.query` Method
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-The suggested way to perform searches is to use the
-:meth:`repoze.catalog.Catalog.query` method. This method accepts a
-number of arguments:
-
-``queryobject``
- A query object or a string representing the query.
-
-``sort_index``
- The name of the index used to sort the results.
-
-``limit``
- Limit the number of results returned to this argument, which should be
- an integer. This is only used if ``sort_index`` is also specified.
-
-``reverse``
- Reverse the order of the result sequence if this is ``True``. Only used
- if ``sort_index`` is also specified.
-
-For example::
-
- from repoze.catalog.catalog import FileStorageCatalogFactory
- from repoze.catalog.catalog import ConnectionManager
- from repoze.catalog.query import Eq
-
- factory = FileStorageCatalogFactory('catalog.db', 'mycatalog')
- manager = ConnectionManager()
- catalog = factory(manager)
- numdocs, results = catalog.query(Eq('flavors', 'peach'))
- print (numdocs, [ x for x in results ])
-
-The results of the above search will search the corpus for documents
-which have a result in the ``flavor`` index that matches the value
-``peach``.
-
-The :meth:`repoze.catalog.Catalog.query` method will return a
-two-tuple, with the first element in the sequence being the length of
-the result set, and the second element being the result set
-itself. Our above example will print::
-
- (1, [1])
-
-The first element in the tuple is the length of the result set (the
-integer ``1``, in this case).
-
-The second element in the tuple is the result set. It has one item.
-This item is the document id for the content we indexed. Your
-application is responsible for resolving this document identifier back
-to its constituent content.
-
-.. warning:: The result set is only guaranteed to be an iterable. It
- will always be of a particular type, and *not* always sliceable;
- for example it may be a generator.
-
-You can also combine query objects, using boolean operations, to search
-multiple indexes:
-
-.. code-block:: python
- :linenos:
-
- from repoze.catalog.catalog import FileStorageCatalogFactory
- from repoze.catalog.catalog import ConnectionManager
-
- factory = FileStorageCatalogFactory('catalog.db', 'mycatalog')
- manager = ConnectionManager()
- catalog = factory(manager)
- numdocs, results = catalog.query(
- Eq('flavors', 'peach') & Eq('texts', 'nutty'))
- print (numdocs, [ x for x in results ])
-
-The results of the above search will return the following::
-
- (0, [])
-
-This is because no results in our index match a document which has
-both a flavor of ``peach`` and text which contains the word ``nutty``.
-
-You can sort the result set using ``sort_index``. The value of
-``sort_index`` should be the name of an index which supports being
-used as a sort index::
-
- from repoze.catalog.query import Range
-
- numdocs, results = catalog.query(
- Range('flavors', 'peach', 'pistachio'),
- sort_index='flavors')
- print (numdocs, [ x for x in results ])
-
-Would result in::
-
- (2, [1, 2])
-
-The default sort order is ascending. You can reverse the sort using
-``reverse``::
-
- from repoze.catalog.query import Range
-
- numdocs, results = catalog.query(
- Range('flavors', 'peach', 'pistachio'),
- sort_index='flavors',
- reverse=True)
- print (numdocs, [ x for x in results ])
-
-Would result in::
-
- (2, [2, 1])
-
-Query Objects
-!!!!!!!!!!!!!
-
-The value passed as the ``queryobject`` argument to
-:meth:`repoze.catalog.Catalog.query` may be one of two distinct types:
-
-- a "raw" :term:`query object`
-
-- a "CQE" string representing a domain-specific-language expression
- which will be used to *generate* a :term:`query object`. "CQE"
- stands for "catalog query expression".
-
-For example, you can construct a raw query object using Python, and
-pass it as ``queryobject`` to the :meth:`repoze.catalog.Catalog.query`
-method:
-
-.. code-block:: python
- :linenos:
-
- from repoze.catalog.query import Eq
- results = catalog.query(Eq('index_name', 'value'))
-
-Or you can allow repoze.catalog to construct a query object on your
-behalf by passing a *string* as ``queryobject``.
-
-.. code-block:: python
- :linenos:
-
- from repoze.catalog.query import Eq
- catalog.query('index_name == "value"')
-
-The above string is a CQE. A "CQE" is a string representing a Python
-expression which uses index names and values. It is parsed by the
-catalog to create a query object.
-
-.. warning:: CQE strings are not supported on Python versions < 2.6.
-
-Whether a query object is used directly or query objects are generated
-as the result of a CQE, an individual query object will be one of two
-types: a comparator or a boolean operator. A comparator performs a single
-query on a single index. A boolean operator allows results from
-individual queries to be combined using boolean operations. For example:
-
-.. code-block:: python
- :linenos:
-
- from repoze.catalog.query import And, Eq, Contains
- query = And(Eq('author', 'crossi'), Contains('body', 'biscuits'))
-
-In the above example, ``And`` is a boolean operator, and both ``Eq`` and
-``Contains`` are comparison operators. The resulting query will search two
-indexes, ``author`` and ``body``. Because the individual comparators are
-passed as arguments to the ``And`` set operator, the result becomes all
-documents which satisfy *both* comparators.
-
-All query objects overload the bitwise and (``&``) and or (``|``) operators
-and can be combined using these. The above query could also have been written
-as follows:
-
-.. code-block:: python
- :linenos:
-
- query = Eq('author', 'crossi') & Contains('body', 'biscuits')
-
-.. note:: Although it would be more intuitive to use the boolean operators,
- ``or`` and ``and`` for this rather than bitwise operators, Python does not
- allow overloading boolean operators.
-
-Query objects may also be created by parsing a :term:`CQE` string.
-The query parser uses Python's internal code parser to parse CQE query
-expression strings, so the syntax is just like Python::
-
- mycatalog.query("author == 'crossi' and 'biscuits' in body")
-
-The query parser allows name substitution in expressions. Names are
-resolved using a dict passed into
-:meth:`repoze.catalog.Catalog.query`::
-
- author = request.params.get("author")
- word = request.params.get("search_term")
- query = mycatalog.query("author == author and word in body",
- names=locals())
-
-Unlike true Python expressions, ordering of the terms in a CQE
-expression is important for comparators. For most comparators the
-``index_name`` must be written on the left. The following, for
-example, would raise an exception::
-
- query = mycatalog.query("'crossi' == author")
-
-Note that not all index types support all comparators. An attempt to
-perform a query using a comparator that is not supported by the index
-being queried will result in a NotImplementedError being raised when
-the query is performed.
-
-Comparators
-!!!!!!!!!!!
-
-The supported comparator operators are as follows:
-
-Equal To
-########
-
-Python::
-
- from repoze.catalog.query import Eq
- Eq(index_name, value)
-
-CQE::
-
- index_name == value
-
-Not Equal To
-############
-
-Python::
-
- from repoze.catalog.query import NotEq
- NotEq(index_name, value)
-
-CQE::
-
- index_name != value
-
-Greater Than
-############
-
-Python::
-
- from repoze.catalog.query import Gt
- Gt(index_name, value)
-
-CQE::
-
- index_name > value
-
-Less Than
-#########
-
-Python::
-
- from repoze.catalog.query import Lt
- Lt(index_name, value)
-
-CQE::
-
- index_name < value
-
-Greater Than Or Equal To
-########################
-
-Python::
-
- from repoze.catalog.query import Ge
- Ge(index_name, value)
-
-CQE::
-
- index_name >= value
-
-Less Than Or Equal To
-#####################
-
-Python::
-
- from repoze.catalog.query import Ge
- Le(index_name, value)
-
-CQE::
-
- index_name <= value
-
-Contains
-########
-
-Python::
-
- from repoze.catalog.query import Contains
- Contains(index_name, value)
-
-CQE::
-
- value in index_name
-
-Does Not Contain
-################
-
-Python::
-
- from repoze.catalog.query import DoesNotContain
- DoesNotContain(index_name, value)
-
-CQE::
-
- value not in index_name
-
-Any
-###
-
-Python::
-
- from repoze.catalog.query import Any
- Any(index_name, [value1, value2, ...])
-
-CQE::
-
- index_name == value1 or index_name == value2 or etc...
- index_name in any([value1, value2, ...])
- index_name in any(values)
-
-Not Any (aka None Of)
-#####################
-
-Python::
-
- from repoze.catalog.query import NotAny
- NotAny(index_name, [value1, value2, ...])
-
-CQE::
-
- index_name != value1 and index_name != value2 and etc...
- index_name not in any([value1, value2, ...])
- index_name not in any(values)
-
-All
-###
-
-Python::
-
- from repoze.catalog.query import All
- All(index_name, [value1, value2, ...])
-
-CQE::
-
- index_name == value1 and index_name == value2 and etc...
- index_name in all([value1, value2, ...])
- index_name in all(values)
-
-Not All
-#######
-
-Python::
-
- from repoze.catalog.query import NotAll
- NotAll(index_name, [value1, value2, ...])
-
-CQE::
-
- index_name != value1 or index_name != value2 or etc...
- index_name not in all([value1, value2, ...])
- index_name not in all(values)
-
-Within Range
-############
-
-Python::
-
- from repoze.catalog.query import InRange
- InRange(index_name, start, end,
- start_exclusive=False, end_exclusive=False)
-
-CQE::
-
- index_name >= start and index_name <= end
- start < index_name < end
-
-Not Within Range
-################
-
-Python::
-
- from repoze.catalog.query import NotInRange
- NotInRange(index_name, start, end,
- start_exclusive=False, end_exclusive=False)
-
-CQE::
-
- index_name <= start or index_name >= end
- not(start < index_name < end)
-
-Boolean Operators
-!!!!!!!!!!!!!!!!!
-
-The following set operators are allowed in queries:
-
-And
-###
-
-Python (explicit)::
-
- from repoze.catalog.query import And
- And(query1, query2)
-
-Python (implicit)::
-
- query1 & query2
-
-CQE::
-
- query1 and query2
- query1 & query2
-
-Or
-##
-
-Python (explicit)::
-
- from repoze.catalog.query import Or
- Or(query1, query2)
-
-Python (implicit)::
-
- query1 | query2
-
-CQE::
-
- query1 or query2
- query1 | query2
-
-Not
-###
-
-Python (explicit)::
-
- from repoze.catalog.query import Not
- Not(query1, query2)
-
-CQE::
-
- not query1
-
-Search Using the :meth:`repoze.catalog.Catalog.search` Method (Deprecated)
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-.. warning::
-
- The :meth:`repoze.catalog.Catalog.search` method is deprecated as of
- :mod:`repoze.catalog` 0.8. Use :meth:`repoze.catalog.Catalog.query`
- instead.
-
-We can pass a query into our catalog's ``search`` method, which is
-composed of the name of our index and a value we'd like to find a
-document for.
-
-.. code-block:: python
- :linenos:
-
- from repoze.catalog.catalog import FileStorageCatalogFactory
- from repoze.catalog.catalog import ConnectionManager
-
- factory = FileStorageCatalogFactory('catalog.db', 'mycatalog')
- manager = ConnectionManager()
- catalog = factory(manager)
- numdocs, results = catalog.search(flavors=('peach', 'peach'))
- print (numdocs, [ x for x in results ])
-
-The results of the above search will search the corpus for documents
-which have a result in the ``flavor`` index that matches the value
-``peach``. Since the index is a "field" index, its query arguments
-are a "range" search: you can read ``('peach', 'peach')`` as "from
-peach to peach". You could say ``('peach', 'pistachio')`` to find all
-documents that are in the "range" from peach to pistachio.
-
-The :meth:`repoze.catalog.Catalog.search` method will return a
-two-tuple, with the first element in the sequence being the length of
-the result set, and the second element being the result set itself.
-Our above example will print:
-
- (1, [1])
-
-The first element in the tuple is the length of the result set (the
-integer ``1``, in this case).
-
-The second element in the tuple is the result set. It has one item.
-This item is the document id for the content we indexed. Your
-application is responsible for resolving this document identifier back
-to its constituent content.
-
-You can also pass compound search parameters for multiple indexes.
-The results are intersected to provide a result:
-
-.. code-block:: python
- :linenos:
-
- from repoze.catalog.catalog import FileStorageCatalogFactory
- from repoze.catalog.catalog import ConnectionManager
-
- factory = FileStorageCatalogFactory('catalog.db', 'mycatalog')
- manager = ConnectionManager()
- catalog = factory(manager)
- numdocs, results = catalog.search(flavors=('peach', 'peach'), texts='nutty')
- print (numdocs, [ x for x in results ])
-
-The results of the above search will return the following:
-
- (0, [])
-
-This is because no results in our index match a document which
-has both a flavor of ``peach`` and text which contains the word
-``nutty``.
-
-See the :term:`zope.index` documentation and implementation for more
-information about what specific index types expect for query
-parameters.
-
-You can also use a field index as a ``sort_index``, which sorts the
-document ids based on the values for that docid present in that index::
-
- numdocs, results = catalog.search(flavors=('peach', 'pistachio'),
- sort_index='flavors')
- print (numdocs, [ x for x in results ])
- (2, [1, 2])
-
-The default sort order is ascending. You can reverse the sort using
-``reverse``::
-
- numdocs, results = catalog.search(flavors=('peach', 'pistachio'),
- sort_index='flavors',
- reverse=True)
- print (numdocs, [ x for x in results ])
- (2, [2, 1])
-
-If you use a sort index, you may choose to limit the number of results
-returned. Do this by passing ``limit`` with an integer value of the
-number of results you want. Note that this parameter has no effect if
-you do not supply a ``sort_index``::
-
- numdocs, results = catalog.search(flavors=('peach', 'pistachio'),
- sort_index='flavors',
- limit=1)
- print (numdocs, [ x for x in results ])
- (1, [1])
-
-You may combine ``reverse`` and ``limit`` as necessary.
-
-If a sort_index is used, and the sort index you're using does not
-contain all the documents returned by the search, the ``numdocs``
-value returned by ``search`` may be incorrect. There will be fewer
-results than those indicated by ``numdocs`` in this circumstance.
-
-When querying a text index, to sort the results by relevance, specify
-the name of the text index as the sort index. The most relevant
-results will be provided first, unless you specify reverse=True, in
-which case the least relevant will be provided first.
-
-Document Map
-------------
-
-An implementation of a "document map" suitable for ZODB applications
-exists within the ``repoze.bfg.document.DocumentMap`` class. A
-document map allows you to map document ids to "addresses" (e.g. paths
-or unique identifiers). See :ref:`api_document_section` in the API
-documentation chapter for more information.
-
-Restrictions
-------------
-
-Values indexed by a :mod:`repoze.catalog` catalog cannot subclass from the
-ZODB ``Persistent`` class. This is a safeguard to ensure that
-irresolveable cross-database references aren't put into the catalog's
-(separate) database.
-
-Gotchas
--------
-
-When the ``ConnectionManager`` 's ``commit`` method is called, it will
-commit a transaction for all databases participating in Zope
-transaction management. Don't use this method if you already have
-transaction management enabled in another way.

0 comments on commit 63e8696

Please sign in to comment.
Something went wrong with that request. Please try again.