New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Search multi indexes for multi-language results #227

Open
xantos05 opened this Issue Feb 3, 2013 · 9 comments

Comments

Projects
None yet
@xantos05

xantos05 commented Feb 3, 2013

I've created a multi-lingual website with Symfony2 using the (Gedmo) Translatable behavior extension for Doctrine2. This works fine but now i'm looking for a way to use the ElasticaBundle to create a nice searchoption. I want German users to search in the German-translation but also in the English translation.

At the moment i'm trying to use separate indexes for each language. My config.yml looks like this:

foq_elastica:    
    clients:
        default: { host: localhost, port: 9200 }    
    indexes:                
        articles_en:
            client:default                                                                                  
            types:
                article:
                mappings:
                    name: { boost: 5, analyzer: my_analyzer }                                                                         
                persistence:
                    driver: orm
                    model:  Test\SiteBundle\Entity\Article
                    identifier: id
                    provider:
                        service: elastica.translation.provider.article.en                        
                    finder:
        articles_de:
        ....
        articles_nl:
        .....

This works fine if you want to search through one index but searching two indexes seems not possible with this bundle or am I wrong?

Is there a way to do this? Any help will be appreciated!

Rick

@kristofvc

This comment has been minimized.

Show comment
Hide comment
@kristofvc

kristofvc Feb 8, 2013

I had kind of the same problem but with projects instead of articles. I just made one index, in your case this would be called "article".
Then in a custom provider I would add an article to your index for every language. The id for the elastic document you add would then be "$article->getId() .'_' . $article->getTranslatableLocale()" (Look at the gedmo translatable extension documentation for this last method).

Then the hard part, mapping your elastica search results to the model (since you got a custom id, you got to add a custom mapper).

Extend the existing mapper class and add the following method:

protected function findByIdentifiers(array $identifierValues, $hydrate)
    {
        if (empty($identifierValues)) {
            return array();
        }

        foreach($identifierValues as $key => $value){
            $identifierValues[$key] = current(explode("_", $value));
        }

        $hydrationMode = $hydrate ? Query::HYDRATE_OBJECT : Query::HYDRATE_ARRAY;
        $qb = $this->registry
            ->getManagerForClass($this->objectClass)
            ->getRepository($this->objectClass)
            ->createQueryBuilder('o');
        /* @var $qb \Doctrine\ORM\QueryBuilder */
        $qb->where($qb->expr()->in('o.'.$this->options['identifier'], ':values'))
            ->setParameter('values', $identifierValues);

        return $qb->getQuery()->setHydrationMode($hydrationMode)->execute();
    }

This will take the array with result-identifiers, and cut the _language from them so you have an array of article-id's.

You then add a service for this:

article.search.elastica_to_model_transformer:
    class: Article
    arguments: [@doctrine, ArticleBundle\Entity\Article, [identifier: id, hydrate: true]]

and put the following in your search-config:

persistence:
            elastica_to_model_transformer:
                    service: article.search.elastica_to_model_transformer

You can then search for articles in multiple languages.

kristofvc commented Feb 8, 2013

I had kind of the same problem but with projects instead of articles. I just made one index, in your case this would be called "article".
Then in a custom provider I would add an article to your index for every language. The id for the elastic document you add would then be "$article->getId() .'_' . $article->getTranslatableLocale()" (Look at the gedmo translatable extension documentation for this last method).

Then the hard part, mapping your elastica search results to the model (since you got a custom id, you got to add a custom mapper).

Extend the existing mapper class and add the following method:

protected function findByIdentifiers(array $identifierValues, $hydrate)
    {
        if (empty($identifierValues)) {
            return array();
        }

        foreach($identifierValues as $key => $value){
            $identifierValues[$key] = current(explode("_", $value));
        }

        $hydrationMode = $hydrate ? Query::HYDRATE_OBJECT : Query::HYDRATE_ARRAY;
        $qb = $this->registry
            ->getManagerForClass($this->objectClass)
            ->getRepository($this->objectClass)
            ->createQueryBuilder('o');
        /* @var $qb \Doctrine\ORM\QueryBuilder */
        $qb->where($qb->expr()->in('o.'.$this->options['identifier'], ':values'))
            ->setParameter('values', $identifierValues);

        return $qb->getQuery()->setHydrationMode($hydrationMode)->execute();
    }

This will take the array with result-identifiers, and cut the _language from them so you have an array of article-id's.

You then add a service for this:

article.search.elastica_to_model_transformer:
    class: Article
    arguments: [@doctrine, ArticleBundle\Entity\Article, [identifier: id, hydrate: true]]

and put the following in your search-config:

persistence:
            elastica_to_model_transformer:
                    service: article.search.elastica_to_model_transformer

You can then search for articles in multiple languages.

@ibasaw

This comment has been minimized.

Show comment
Hide comment
@ibasaw

ibasaw Jun 24, 2013

is there a simple way to do this ?

In ES, you can search on multiples indexes /types at same time.

ibasaw commented Jun 24, 2013

is there a simple way to do this ?

In ES, you can search on multiples indexes /types at same time.

@gagarine

This comment has been minimized.

Show comment
Hide comment
@gagarine

gagarine Mar 17, 2014

Contributor

Would be cool to have some kind of pointer about the way to go...

Contributor

gagarine commented Mar 17, 2014

Would be cool to have some kind of pointer about the way to go...

@merk merk added this to the 3.1.0 milestone Jul 12, 2014

@topwebstudio

This comment has been minimized.

Show comment
Hide comment
@topwebstudio

topwebstudio Aug 4, 2014

Contributor

+1 on this, better solution for searching translatable entities would be much appriciated

Contributor

topwebstudio commented Aug 4, 2014

+1 on this, better solution for searching translatable entities would be much appriciated

@igormancos

This comment has been minimized.

Show comment
Hide comment
@igormancos

igormancos Sep 23, 2014

+1, wait better solution

igormancos commented Sep 23, 2014

+1, wait better solution

@raistlin

This comment has been minimized.

Show comment
Hide comment
@raistlin

raistlin Aug 5, 2015

+1, Still no solution here?

raistlin commented Aug 5, 2015

+1, Still no solution here?

@coudenysj

This comment has been minimized.

Show comment
Hide comment
@coudenysj

coudenysj Aug 5, 2015

The problem of working with a custom provider, is that the listener is not working anymore.

A better solution would be to write a custom ObjectPersister and let the persister to the "duplication". The elastica_to_model_transformer is still necessary, because you need composite ES identifiers.

coudenysj commented Aug 5, 2015

The problem of working with a custom provider, is that the listener is not working anymore.

A better solution would be to write a custom ObjectPersister and let the persister to the "duplication". The elastica_to_model_transformer is still necessary, because you need composite ES identifiers.

@maxpou

This comment has been minimized.

Show comment
Hide comment
@maxpou

maxpou Feb 10, 2016

Hi,
I got the same problem.
So, I choose an other service : fos_elastica.client. Anyway, I had no choice because my finders didn't work...

Here's a part of my solution:

use Elastica\Request as ElasticaRequest;

//...

$userQuery = "bob";
$client = $this->container->get('fos_elastica.client');
$query = array(
    'query' => array(
        'query_string' => array(
            'query' => $userQuery,
        )
    ),
    'from' => '6',  //optional (default 0)
    'size' => '5'   //optional (default 10)
);

$results = $client->request('/myIndex/_search', ElasticaRequest::GET, $query);
$results->getData(); //Elasticsearch JSON response

Further more reading : http://elastica.io/examples/

maxpou commented Feb 10, 2016

Hi,
I got the same problem.
So, I choose an other service : fos_elastica.client. Anyway, I had no choice because my finders didn't work...

Here's a part of my solution:

use Elastica\Request as ElasticaRequest;

//...

$userQuery = "bob";
$client = $this->container->get('fos_elastica.client');
$query = array(
    'query' => array(
        'query_string' => array(
            'query' => $userQuery,
        )
    ),
    'from' => '6',  //optional (default 0)
    'size' => '5'   //optional (default 10)
);

$results = $client->request('/myIndex/_search', ElasticaRequest::GET, $query);
$results->getData(); //Elasticsearch JSON response

Further more reading : http://elastica.io/examples/

@tuxado

This comment has been minimized.

Show comment
Hide comment
@tuxado

tuxado Apr 16, 2016

i have use nested to translations like that come directly from the entity :

            book:
                mappings:
                    translations:
                        type: nested
                        properties :
                            name: ~
                            locale: ~
                            description: ~

result

{
"_index": "index",
"_type": "book",
"_id": "1",
"_version": 1,
"_score": 1,
"_source": {
"translations": [
{
"name": "....",
"locale": "en",
"description": "...."
}
],
....
.....
and i use the nested query to search using bool with must locale : "en"

tuxado commented Apr 16, 2016

i have use nested to translations like that come directly from the entity :

            book:
                mappings:
                    translations:
                        type: nested
                        properties :
                            name: ~
                            locale: ~
                            description: ~

result

{
"_index": "index",
"_type": "book",
"_id": "1",
"_version": 1,
"_score": 1,
"_source": {
"translations": [
{
"name": "....",
"locale": "en",
"description": "...."
}
],
....
.....
and i use the nested query to search using bool with must locale : "en"

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment