Elasticsearch filter

[dkarlovi]

It would maybe make sense to ship with a out-of-the-box advanced full-text search & filter capability, obvious candidate being Elasticsearch with FOSElasticaBundle.

Seeing as the data provider is still ORM, only the filtering is done with Elastic (which gets populated and updated out-of-band), I guess it would just be a custom filter (applying ID-based filter and order), not a full blown data provider? Would this make sense in this context?

[sstok]

Note I plan to add a Api-Platform bridge for RollerworksSearch very soon, unless someone is able to help :) rollerworks/search#87 (comment)

Processing is actually provided now, but the https://github.com/rollerworks/search/blob/master/src/Exception/InvalidSearchConditionException.php needs to be cast to an Error context for Hydra/Swagger or whatever it's called 😄 that would be first start. And you need something to configure the data provider (of Api Platform).

ElasticSearch support is planned, and next on my list of things to do.

[dunglas]

@sstok can't api-platform/docs#186 do the trick for the exception?

[sstok]

The exception throw by RollerworksSearch input-processor can contain a list of error messages (similar to the Symfony Validation Violations).

https://github.com/api-platform/core/blob/master/src/Hydra/Serializer/ConstraintViolationListNormalizer.php the basis of this should work 😃 I will try to experiment with this.

[sstok]

It's a start 😄 https://github.com/rollerworks/search-api-platform

[soyuka]

@sstok I made an Oracle Text extension that helps creating indexes and build search queries with CONTAINS. Lmk if you'd be interested in such a thing ;).

[sstok]

@soyuka I haven't used/tested Oracle, but there is a Doctrine DBAL/ORM extension for the search system:

https://github.com/rollerworks/search-doctrine-dbal
https://github.com/rollerworks/search-doctrine-orm

[soyuka]

Yeah I saw those, this is why I mentioned oracle text! I think that modern DBMS all have some kind of optimized way of doing text search, sometimes there's no need to use ES on top.

[dkarlovi]

@soyuka yes, but those are often very basic, MySQL has basic search too.

ES gives you access to quite advanced features you most often wont get from these built-in implementations.

[soyuka]

Hmm, you mean MariaDb though, mysql has been bought by oracle \o/.

Of course that ES gives you more advances features, it has been built for search. Though it's often enough to just use the DMBS:

• postgresql
• oracle
• mariadb

Both postgres and oracle have nice indexes with features like stop words and fuzzy matches :). Most of the time those are enough IMHO.

[dkarlovi]

Nope, I mean MySQL, doesn't matter Oracle bought 'em, "Oracle" in this context means "Oracle DB" which isn't the same thing as Oracle's MySQL offering.

Side note: Oracle actually bought Sun who bought MySQL AB previously, BTW, I still don't call MySQL "Sun". :) None of this is relevant to the issue in hand which is "advanced search features for API Platform".

[sstok]

Amazing news! API-Platform support for RollerworksSearch is ready (for testing).
Note that ElasticSearch is currently not yet supported, but Doctrine ORM is.

RollerworksSearch 2.0 as a whole is still in alpha phase because I want to be sure that everything is taken care of properly. But most parts are heavily tested.

Installing

Integration for the Symfony FrameworkBundle is completely provided, install the following:

• rollerworks/search-bundle (>=v2.0.0-ALPHA5)
• rollerworks/search-doctrine-orm (>=v2.0.0-ALPHA2)
• rollerworks/search-api-platform (>=0.2.0)

And enable the RollerworksSearchBundle, bundle configuration is fully automatic 👍 (unless you use a custom entity-manager)

Metadata

The integration bridge supports multiple contexts for a Resource (backend/frontend) each with there own configuration, a _defaults context allows to define shared values for all contexts (similar to Symfony DI _defaults logic).

Unless an context is provided at Request#attibutes _api_search_context context _any is used.

To make your Resource searchable you need to set the rollerworks_search.contexts in the Resource Metadata attributes. Unsupported Resources are simple ignored.

/**
 * A book.
 *
 * @see http://schema.org/Book Documentation on Schema.org
 *
 * @ORM\Entity
 * @ApiResource(
 *     iri="http://schema.org/Book",
 *     attributes={
 *         "rollerworks_search"={
 *             "contexts"={
 *                 "_defaults"={
 *                      "fieldset" = "Acme\AppBundle\Search\FieldSet\BookFieldSet"
 *                  },
 *                  "_any"={
 *                      "doctrine_orm"={
 *                          "mappings"={
 *                              "id" = "id",
 *                              "title" = "title"
 *                          }
 *                      }
 *                  }
 *             }
 *         }
 *     }
 * )
 */

A FieldSet holds which fields can be used for searching (independent from the Data provider).
http://rollerworkssearch.readthedocs.io/en/latest/introduction.html#fieldset

The doctrine_orm configuration is to map a search field to a property and set-up relations for child entities. Full example:

"doctrine_orm" = {
                "relations" = {
                    "alias" = { "type" = "(left | right | inner)", "entity" "join", "conditionType" = null, "condition" = null, "indexBy" = null }
                },
                "mappings" = {
                    "mapping-name" = { "property" = "...", "alias" = "...", "db_type" = null }
                },
            },

When the mapping is a single value it's assumed to be a property-name, else it need a property key. The alias defaults to 'o' (root entity). db_type is required when it cannot be auto guessed.

Relations consists of an alias, a join and entity (both are required), type defaults to left.
They are currently only used for searching (not for fetching, I'm not sure if this should be done here or else were).

See https://github.com/rollerworks/search-api-platform/blob/master/Metadata-reference.md for a complete reference. Proper documentation is currently pending.

Searching

To perform a search, supply the search condition as array format in the URL. Eg.
http://127.0.0.1:8000/books.json?search[fields][id][simple-values][0]=1&search[fields][id][simple-values][1]=2

It's best to use the RollerworksSearch core library (ConditonBuilder) as SDK to compose a SearchCondition for processing.

Note: When the Condition contains duplicate/redundant values a redirect is issued with the new condition.

Processed conditions and generated DQL conditions are cached (unless caching is disabled) for future requests.

RFC

If you have any comments or idea's please let me know 👍 ❤️

I'm not sure if the relation configuration is as clean as it can be, and it's possible there can be conflicts with eager loading. So please try as much as you can.

ElasticSearch is something I hope to start working on really soon now, and of course integration will be provided for the API-Platform bridge 😄

[soyuka]

Could you explain what does RollerworksSearch do that the embedded filters can't do?

Great work btw, looks nice!

[sstok]

RollerworksSearch is a complete search solution, it takes care of everything.

Processing the Input (transforming to a Model format), optimizing the condition (supports nested grouping), and for Doctrine ORM you have some special options that allow for complex queries (like searching for a user based on both date and age (in years)). And handling relations without to much complexity, it does require some manual set-up but this allows for greater flexibility and less problems.

But what's most interesting is the build-in caching (for best performance) and error handling (similar to validation Constraints). Plus, you can reuse the existing Search system if you have an API/classic server processing set-up.

And in the future I plan to add support for smarty-query rollerworks/search#23 and a client-side condition builder.

You use this search system when you actually need to search for something rather then limiting something based on a (simple) filter 👍