graphene-elastic

Elasticsearch (DSL)/ OpenSearch (DSL) integration for Graphene.

Prerequisites

• Graphene 2.x. Support for Graphene 1.x is not intended.
• Python 3.6, 3.7, 3.8, 3.9 and 3.10. Support for Python 2 is not intended.
• Elasticsearch 6.x, 7.x. Support for Elasticsearch 5.x is not intended.
• OpenSearch 1.x, 2.x.

Main features and highlights

• Implemented ElasticsearchConnectionField and ElasticsearchObjectType are the core classes to work with graphene.
• Pluggable backends for searching, filtering, ordering, etc. Don't like existing ones? Override, extend or write your own.
• Search backend.
• Filter backend.
• Ordering backend.
• Pagination.
• Highlighting backend.
• Source filter backend.
• Faceted search backend (including global aggregations).
• Post filter backend.
• Score filter backend.
• Query string backend.
• Simple query string backend.

See the Road-map for what's yet planned to implemented.

Do you need a similar tool for Django REST Framework? Check django-elasticsearch-dsl-drf.

Demo

Check the live demo app (FastAPI + Graphene 2 + Elasticsearch 7) hosted on Heroku and bonsai.io.

Documentation

Documentation is available on Read the Docs.

Installation

Install latest stable version from PyPI:

pip install graphene-elastic

Or latest development version from GitHub:

pip install https://github.com/barseghyanartur/graphene-elastic/archive/master.zip

Note

Staring from version 0.8, the elasticsearch and elasticsearch-dsl packages are no longer installed by default. You must either install them explicitly in your requirements or install as optional dependencies as follows: pip install graphene-elastic[elasticsearch]. Alternatively, you can use opensearch-py and opensearch-dsl. You would then need to install the opensearch-py and opensearch-dsl packages explicitly in your requirements or install them as optional dependencies as follows: pip install graphene-elastic[opensearch].

Examples

Note

In the examples, we use elasticsearch_dsl package for schema definition. You can however use opensearch_dsl or if you want to achieve portability between Elasticsearch and OpenSearch, use anysearch package. Read more here.

Install requirements

pip install -r requirements.txt

Populate sample data

The following command will create indexes for User and Post documents and populate them with sample data:

./scripts/populate_elasticsearch_data.sh

Sample document definition

search_index/documents/post.py

See examples/search_index/documents/post.py for full example.

import datetime
from elasticsearch_dsl import (
    Boolean,
    Date,
    Document,
    InnerDoc,
    Keyword,
    Nested,
    Text,
    Integer,
)

class Comment(InnerDoc):

    author = Text(fields={'raw': Keyword()})
    content = Text(analyzer='snowball')
    created_at = Date()

    def age(self):
        return datetime.datetime.now() - self.created_at


class Post(Document):

    title = Text(
        fields={'raw': Keyword()}
    )
    content = Text()
    created_at = Date()
    published = Boolean()
    category = Text(
        fields={'raw': Keyword()}
    )
    comments = Nested(Comment)
    tags = Text(
        analyzer=html_strip,
        fields={'raw': Keyword(multi=True)},
        multi=True
    )
    num_views = Integer()

    class Index:
        name = 'blog_post'
        settings = {
            'number_of_shards': 1,
            'number_of_replicas': 1,
            'blocks': {'read_only_allow_delete': None},
        }

Sample apps

Sample Flask app

Run the sample Flask app:

./scripts/run_flask.sh

Open Flask graphiql client

http://127.0.0.1:8001/graphql

Sample Django app

Run the sample Django app:

./scripts/run_django.sh runserver

Open Django graphiql client

http://127.0.0.1:8000/graphql

ConnectionField example

ConnectionField is the most flexible and feature rich solution you have. It uses filter backends which you can tie to your needs the way you want in a declarative manner.

Sample schema definition

import graphene
from graphene_elastic import (
    ElasticsearchObjectType,
    ElasticsearchConnectionField,
)
from graphene_elastic.filter_backends import (
    FilteringFilterBackend,
    SearchFilterBackend,
    HighlightFilterBackend,
    OrderingFilterBackend,
    DefaultOrderingFilterBackend,
)
from graphene_elastic.constants import (
    LOOKUP_FILTER_PREFIX,
    LOOKUP_FILTER_TERM,
    LOOKUP_FILTER_TERMS,
    LOOKUP_FILTER_WILDCARD,
    LOOKUP_QUERY_EXCLUDE,
    LOOKUP_QUERY_IN,
)

# Object type definition
class Post(ElasticsearchObjectType):

    class Meta(object):
        document = PostDocument
        interfaces = (Node,)
        filter_backends = [
            FilteringFilterBackend,
            SearchFilterBackend,
            HighlightFilterBackend,
            OrderingFilterBackend,
            DefaultOrderingFilterBackend,
        ]

        # For `FilteringFilterBackend` backend
        filter_fields = {
            # The dictionary key (in this case `title`) is the name of
            # the corresponding GraphQL query argument. The dictionary
            # value could be simple or complex structure (in this case
            # complex). The `field` key points to the `title.raw`, which
            # is the field name in the Elasticsearch document
            # (`PostDocument`). Since `lookups` key is provided, number
            # of lookups is limited to the given set, while term is the
            # default lookup (as specified in `default_lookup`).
            'title': {
                'field': 'title.raw',
                # Available lookups
                'lookups': [
                    LOOKUP_FILTER_TERM,
                    LOOKUP_FILTER_TERMS,
                    LOOKUP_FILTER_PREFIX,
                    LOOKUP_FILTER_WILDCARD,
                    LOOKUP_QUERY_IN,
                    LOOKUP_QUERY_EXCLUDE,
                ],
                # Default lookup
                'default_lookup': LOOKUP_FILTER_TERM,
            },

            # The dictionary key (in this case `category`) is the name of
            # the corresponding GraphQL query argument. Since no lookups
            # or default_lookup is provided, defaults are used (all lookups
            # available, term is the default lookup). The dictionary value
            # (in this case `category.raw`) is the field name in the
            # Elasticsearch document (`PostDocument`).
            'category': 'category.raw',

            # The dictionary key (in this case `tags`) is the name of
            # the corresponding GraphQL query argument. Since no lookups
            # or default_lookup is provided, defaults are used (all lookups
            # available, term is the default lookup). The dictionary value
            # (in this case `tags.raw`) is the field name in the
            # Elasticsearch document (`PostDocument`).
            'tags': 'tags.raw',

            # The dictionary key (in this case `num_views`) is the name of
            # the corresponding GraphQL query argument. Since no lookups
            # or default_lookup is provided, defaults are used (all lookups
            # available, term is the default lookup). The dictionary value
            # (in this case `num_views`) is the field name in the
            # Elasticsearch document (`PostDocument`).
            'num_views': 'num_views',
        }

        # For `SearchFilterBackend` backend
        search_fields = {
            'title': {'boost': 4},
            'content': {'boost': 2},
            'category': None,
        }

        # For `OrderingFilterBackend` backend
        ordering_fields = {
            # The dictionary key (in this case `tags`) is the name of
            # the corresponding GraphQL query argument. The dictionary
            # value (in this case `tags.raw`) is the field name in the
            # Elasticsearch document (`PostDocument`).
            'title': 'title.raw',

            # The dictionary key (in this case `created_at`) is the name of
            # the corresponding GraphQL query argument. The dictionary
            # value (in this case `created_at`) is the field name in the
            # Elasticsearch document (`PostDocument`).
            'created_at': 'created_at',

            # The dictionary key (in this case `num_views`) is the name of
            # the corresponding GraphQL query argument. The dictionary
            # value (in this case `num_views`) is the field name in the
            # Elasticsearch document (`PostDocument`).
            'num_views': 'num_views',
        }

        # For `DefaultOrderingFilterBackend` backend
        ordering_defaults = (
            '-num_views',  # Field name in the Elasticsearch document
            'title.raw',  # Field name in the Elasticsearch document
        )

        # For `HighlightFilterBackend` backend
        highlight_fields = {
            'title': {
                'enabled': True,
                'options': {
                    'pre_tags': ["<b>"],
                    'post_tags': ["</b>"],
                }
            },
            'content': {
                'options': {
                    'fragment_size': 50,
                    'number_of_fragments': 3
                }
            },
            'category': {},
        }

# Query definition
class Query(graphene.ObjectType):
    all_post_documents = ElasticsearchConnectionField(Post)

# Schema definition
schema = graphene.Schema(query=Query)

Filter

Sample queries

Since we didn't specify any lookups on category, by default all lookups are available and the default lookup would be term. Note, that in the {value:"Elastic"} part, the value stands for default lookup, whatever it has been set to.

query PostsQuery {
  allPostDocuments(filter:{category:{value:"Elastic"}}) {
    edges {
      node {
        id
        title
        category
        content
        createdAt
        comments
      }
    }
  }
}

But, we could use another lookup (in example below - terms). Note, that in the {terms:["Elastic", "Python"]} part, the terms is the lookup name.

query PostsQuery {
  allPostDocuments(
        filter:{category:{terms:["Elastic", "Python"]}}
    ) {
    edges {
      node {
        id
        title
        category
        content
        createdAt
        comments
      }
    }
  }
}

Or apply a gt (range) query in addition to filtering:

{
  allPostDocuments(filter:{
        category:{term:"Python"},
        numViews:{gt:"700"}
    }) {
    edges {
      node {
        category
        title
        comments
        numViews
      }
    }
  }
}

Implemented filter lookups

The following lookups are available:

• contains
• ends_with (or endsWith for camelCase)
• exclude
• exists
• gt
• gte
• in
• is_null (or isNull for camelCase)
• lt
• lte
• prefix
• range
• starts_with (or startsWith for camelCase)
• term
• terms
• wildcard

See dedicated documentation on filter lookups for more information.

Search

Search in all fields:

query {
  allPostDocuments(
    search:{query:"Release Box"}
  ) {
    edges {
      node {
        category
        title
        content
      }
    }
  }
}

Search in specific fields:

query {
  allPostDocuments(
    search:{
        title:{value:"Release", boost:2},
        content:{value:"Box"}
    }
  ) {
    edges {
      node {
        category
        title
        content
      }
    }
  }
}

Ordering

Possible choices are ASC and DESC.

query {
  allPostDocuments(
        filter:{category:{term:"Photography"}},
        ordering:{title:ASC}
    ) {
    edges {
      node {
        category
        title
        content
        numViews
        tags
      }
    }
  }
}

Pagination

The first, last, before and after arguments are supported. By default number of results is limited to 100.

query {
  allPostDocuments(first:12) {
    pageInfo {
      startCursor
      endCursor
      hasNextPage
      hasPreviousPage
    }
    edges {
      cursor
      node {
        category
        title
        content
        numViews
      }
    }
  }
}

Highlighting

Simply, list the fields you want to highlight. This works only in combination with search.

query {
  allPostDocuments(
        search:{content:{value:"alice"}, title:{value:"alice"}},
        highlight:[category, content]
    ) {
    edges {
      node {
        title
        content
        highlight
      }
      cursor
    }
  }
}

Road-map

Road-map and development plans.

This package is designed after django-elasticsearch-dsl-drf and is intended to offer similar functionality.

Lots of features are planned to be released in the upcoming Beta releases:

• Suggester backend.
• Nested backend.
• Geo-spatial backend.
• Filter lookup geo_bounding_box (or geoBoundingBox for camelCase).
• Filter lookup geo_distance (or geoDistance for camelCase).
• Filter lookup geo_polygon (or geoPolygon for camelCase).
• More-like-this backend.

Stay tuned or reach out if you want to help.

Testing

Project is covered with tests.

Testing with Docker

make docker-test

Running tests with virtualenv or tox

By defaults tests are executed against the Elasticsearch 7.x.

Run Elasticsearch 7.x with Docker

docker-compose up elasticsearch

Install test requirements

pip install -r requirements/test.txt

To test with all supported Python versions type:

tox

To test against specific environment, type:

tox -e py38-elastic7

To test just your working environment type:

./runtests.py

To run a single test module in your working environment type:

./runtests.py src/graphene_elastic/tests/test_filter_backend.py

To run a single test class in a given test module in your working environment type:

./runtests.py src/graphene_elastic/tests/test_filter_backend.py::FilterBackendElasticTestCase

Debugging

For development purposes, you could use the flask app (easy to debug). Standard pdb works (import pdb; pdb.set_trace()). If ipdb does not work well for you, use ptpdb.

Writing documentation

Keep the following hierarchy.

=====
title
=====

header
======

sub-header
----------

sub-sub-header
~~~~~~~~~~~~~~

sub-sub-sub-header
^^^^^^^^^^^^^^^^^^

sub-sub-sub-sub-header
++++++++++++++++++++++

sub-sub-sub-sub-sub-header
**************************

License

GPL-2.0-only OR LGPL-2.1-or-later

Support

For any security issues contact me at the e-mail given in the Author section. For overall issues, go to GitHub.

Author

Artur Barseghyan <artur.barseghyan@gmail.com>