Skip to content

Commit

Permalink
Updating documentation. (End of day.)
Browse files Browse the repository at this point in the history
  • Loading branch information
Pat Daburu committed Feb 12, 2019
1 parent f2a265b commit 9f176a9
Show file tree
Hide file tree
Showing 5 changed files with 116 additions and 120 deletions.
41 changes: 41 additions & 0 deletions docs/source/blobbing.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
.. _blobbing:

.. image:: _static/images/logo.svg
:width: 100px
:alt: elastalk
:align: right

.. toctree::
:glob:


"Blobbing"
==========

In order to minimize database overhead, some applications may want to store non-searchable document
content in binary form in a field called `blob` so once a document has been indexed.

If you want to store some (or all) of your seed data as a single
`base-64 <https://www.base64decode.org/>`_ `BLOB <https://techterms.com/definition/blob>`_, you can
add a `blobs` key to your :ref:`index configuration file <seed_data_extra_config>`.

Within the `blobs` key you can supply two directives:

:enabled: *(bool)* `true` to convert seed data to BLOBs

:exclude: *(list)* a list of top-level keys in your seed data document that should not be
included in the BLOB

TODO: Convert to TOML.

.. code-block:: javascript
{
"blobs": {
"enabled": true,
"exclude": [
"firstName",
"lastName"
]
}
}
69 changes: 69 additions & 0 deletions docs/source/configuration.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,69 @@
.. _configuration:

.. image:: _static/images/logo.svg
:width: 100px
:alt: elastalk
:align: right

.. toctree::
:glob:


Configuring Your Connection
===========================

Configuration from TOML
-----------------------

In addition to :ref:`configuring from objects <elastalk_flask_config_from_objects>`, you can also
configure `elastalk` connections using `TOML <https://pypi.org/project/toml/>`_.

TOML aims to be a minimal configuration file format that's easy to read due to obvious
semantics. TOML is designed to map unambiguously to a
`hash table <https://en.wikipedia.org/wiki/Hash_table>`_.

-- the TOML project's `README.md <https://github.com/toml-lang/toml>`_

A Sample TOML Configuration
^^^^^^^^^^^^^^^^^^^^^^^^^^^

.. code-block:: ini
[blobs]
excluded = ["owner_", "group_"]
[indexes.cats]
mappings = "cats/mappings.json"
[indexes.dogs.blobs]
enabled = True
excluded = ["name", "breed"]
Options
^^^^^^^

blobs
=====

This section contains global configuration options that control how, when, and which data is
converted to binary representations (see :ref:`Blobbing <blobbing>`).

:excluded: the names of attributes that are never included in binary representations when a
document is packed using the
:py:func:`ElastalkConnection.pack() <elastalk.connect.ElastalkConnection.pack>`
method

TODO

indexes
=======

TODO


Configuration from Objects
--------------------------

If you're wanting to configure your connection from a python object, you're likely using
:ref:`Flask <elastalk_and_flask>`. There is another article on that subject called
:ref:`elastalk_flask_config_from_objects`.
49 changes: 0 additions & 49 deletions docs/source/flask.rst
Original file line number Diff line number Diff line change
Expand Up @@ -78,52 +78,3 @@ settings from an object.



Configuration from TOML
-----------------------

In addition to :ref:`configuring from objects <elastalk_flask_config_from_objects>`, you can also
configure `elastalk` connections using `TOML <https://pypi.org/project/toml/>`_.

TOML aims to be a minimal configuration file format that's easy to read due to obvious
semantics. TOML is designed to map unambiguously to a
`hash table <https://en.wikipedia.org/wiki/Hash_table>`_.

-- the TOML project's `README.md <https://github.com/toml-lang/toml>`_

A Sample TOML Configuration
^^^^^^^^^^^^^^^^^^^^^^^^^^^

.. code-block:: ini
[blobs]
excluded = ["owner_", "group_"]
[indexes.cats]
mappings = "cats/mappings.json"
[indexes.dogs.blobs]
enabled = True
excluded = ["name", "breed"]
Options
^^^^^^^

blobs
=====

This section contains global configuration options that control how, when, and which data is
converted to binary representations (*"BLOBS"*).

:excluded: the names of attributes that are never included in binary representations when a
document is packed using the
:py:func:`ElastalkConnection.pack() <elastalk.connect.ElastalkConnection.pack>`
method

TODO

indexes
=======

TODO


4 changes: 3 additions & 1 deletion docs/source/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -17,11 +17,13 @@ Simple Conveniences for Talking to `Elasticsearch <https://www.elastic.co/produc
:caption: Contents:

getting_started
configuration
seed_data
blobbing
flask
versions
cli
api
seed_data
development
requirements

Expand Down
73 changes: 3 additions & 70 deletions docs/source/seed_data.rst
Original file line number Diff line number Diff line change
Expand Up @@ -75,51 +75,19 @@ represent the individual documents that will be indexed. The name of the file w
`id <https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-id-field.html>`_ of
the document.

.. _seed_data_special_fields:

Special Fields
--------------

By convention, API seed data documents will often specify an `owner_` field and a `group_` field.
In both cases the value should be a string representation of a
`UUID <https://en.wikipedia.org/wiki/Universally_unique_identifier>`_. These are not part
of the actual document but are used by applications to track stored documents.

If your data doesn't need these values, just omit one or both of them.

.. code-block:: python
:linenos:
:emphasize-lines: 2,3
{
"owner_": "3ab2b4e5-6992-43f0-851e-d1f42262cd55",
"group_": "5bc9ad2d-7f72-452c-b32a-d699b20d2b1f",
"timestamp_utc": "2018-12-10 16:36:05",
"location": "/input/directory/data/in/arch/data.zip",
"stats": {
"name": "data.zip",
"created_utc": "2018-10-12 10:56:39",
"modified_utc": "2018-10-12 10:49:27",
"size": 150044002
},
"metadata": {},
"data_sources": [...]
}
.. _seed_data_extra_config:

Extra Configuration
-------------------

You can supply additional information about the seed data in an index by supplying a `config.json`
You can supply additional information about the seed data in an index by supplying a `config.toml`
file in the :ref:`seed_data_indexes` directory.

.. note::

The :py:func:`seed <elastalk.seed.seed>` function supports a parameter called
`config_filename` if, for some reason, you have a reason not to call your configuration files
`"config.json"`.
`config` if, for some reason, you have a reason not to call your configuration files
`"config.toml"`.


.. _seed_data_mappings:
Expand Down Expand Up @@ -180,38 +148,3 @@ Elasticsearch...
}
}
}
"Blobbing"
^^^^^^^^^^

In order to minimize database overhead, some applications may want to store non-searchable document
content in binary form in a field called `blob` so once a document has been indexed, if you inspect
it within the index (for example, using `Kibana <https://www.elastic.co/products/kibana>`_) you may
notice that the only visible fields are the :ref:`special fields <seed_data_special_fields>`.

If you want to store some (or all) of your seed data as a single
`base-64 <https://www.base64decode.org/>`_ `BLOB <https://techterms.com/definition/blob>`_, you can
add a `blobs` key to your :ref:`index configuration file <seed_data_extra_config>`.

Within the `blobs` key you can supply two directives:

:enabled: *(bool)* `true` to convert seed data to BLOBs

:exclude: *(list)* a list of top-level keys in your seed data document that should not be
included in the BLOB

.. code-block:: javascript
{
"blobs": {
"enabled": true,
"exclude": [
"firstName",
"lastName"
]
}
}
.. note::

:ref:`seed_data_special_fields` are always excluded from BLOBs.

0 comments on commit 9f176a9

Please sign in to comment.