New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added a simple search engine #2543

Merged
merged 5 commits into from Nov 19, 2016

Conversation

Projects
None yet
4 participants
@nicosomb
Member

nicosomb commented Nov 4, 2016

Q A
Bug fix? no
New feature? yes
BC breaks? no
Deprecations? no
Tests pass? no
Documentation no
Translation no
Fixed tickets #18
License MIT

Fix #18

I don't know if we'll have time to implement Elasticsearch. That's why I worked on a simple search engine (just a query on the title and the content of the articles).

Todo

  • tests
  • documentation
  • translations
  • when I'm on a category (for example, starred entries), if I do a search, the search engine must filter the starred entries only.
  • add the search form on baggy theme
@tcitworld

This comment has been minimized.

Show comment
Hide comment
@tcitworld

tcitworld Nov 4, 2016

Member

when I'm on a category (for example, starred entries), if I do a search, the search engine must filter the starred entries only.

Same goes for filters if possible.

Member

tcitworld commented Nov 4, 2016

when I'm on a category (for example, starred entries), if I do a search, the search engine must filter the starred entries only.

Same goes for filters if possible.

@nicosomb

This comment has been minimized.

Show comment
Hide comment
@nicosomb

nicosomb Nov 6, 2016

Member

Same goes for filters if possible.

As requested in #2533.

Member

nicosomb commented Nov 6, 2016

Same goes for filters if possible.

As requested in #2533.

Show outdated Hide outdated src/Wallabag/CoreBundle/Repository/EntryRepository.php
->andWhere('e.content LIKE :term')->setParameter('term', '%'.$term.'%')
->orWhere('e.title LIKE :term')->setParameter('term', '%'.$term.'%')
->leftJoin('e.tags', 't')
->groupBy('e.id')

This comment has been minimized.

@j0k3r

j0k3r Nov 6, 2016

Member

I guess this won't work on PosgreSQL because you need to use all the selected fields in the group clause or use a different type of query.

@j0k3r

j0k3r Nov 6, 2016

Member

I guess this won't work on PosgreSQL because you need to use all the selected fields in the group clause or use a different type of query.

This comment has been minimized.

@nicosomb

nicosomb Nov 10, 2016

Member

@j0k3r what about getBuilderForUntaggedByUser where we already have this groupBy?

@nicosomb

nicosomb Nov 10, 2016

Member

@j0k3r what about getBuilderForUntaggedByUser where we already have this groupBy?

@nicosomb

This comment has been minimized.

Show comment
Hide comment
@nicosomb

nicosomb Nov 10, 2016

Member

translations
when I'm on a category (for example, starred entries), if I do a search, the search engine must filter the starred entries only

Member

nicosomb commented Nov 10, 2016

translations
when I'm on a category (for example, starred entries), if I do a search, the search engine must filter the starred entries only

Show outdated Hide outdated src/Wallabag/CoreBundle/Resources/translations/messages.fr.yml
@@ -155,6 +155,7 @@ entry:
archived: "Articles lus"
filtered: "Articles filtrés"
filtered_tags: "Articles filtrés par tags :"
filtered_search: 'Articles filtrés par une recherche :'

This comment has been minimized.

@tcitworld

tcitworld Nov 18, 2016

Member

une ?

@tcitworld

This comment has been minimized.

Show comment
Hide comment
@tcitworld

tcitworld Nov 18, 2016

Member
  • Add keyboard shortcut to focus search field.
Member

tcitworld commented Nov 18, 2016

  • Add keyboard shortcut to focus search field.
@nicosomb

This comment has been minimized.

Show comment
Hide comment
@nicosomb

nicosomb Nov 18, 2016

Member

Shortcut added.

Member

nicosomb commented Nov 18, 2016

Shortcut added.

Show outdated Hide outdated app/Resources/static/themes/baggy/js/shortcuts/main.js
@@ -0,0 +1,8 @@
$(document).ready(() => {

This comment has been minimized.

@j0k3r

j0k3r Nov 18, 2016

Member

This line isn't necessary

@j0k3r

j0k3r Nov 18, 2016

Member

This line isn't necessary

Show outdated Hide outdated app/Resources/static/themes/material/js/shortcuts/main.js
@@ -28,8 +28,14 @@ $(document).ready(() => {
toggleFocus(card);
/* Actions */
Mousetrap.bind('g n', () => {
Mousetrap.bind('g n', (e) => {

This comment has been minimized.

@j0k3r

j0k3r Nov 18, 2016

Member

Does that e is really necessary since you don't use it in the function?

@j0k3r

j0k3r Nov 18, 2016

Member

Does that e is really necessary since you don't use it in the function?

This comment has been minimized.

@tcitworld

tcitworld Nov 18, 2016

Member

Nope, remove. :-)

@tcitworld

tcitworld Nov 18, 2016

Member

Nope, remove. :-)

@nicosomb nicosomb merged commit 6f85bed into 2.2 Nov 19, 2016

2 checks passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details
zappr/pr/specification PR has passed specification checks

@nicosomb nicosomb deleted the search-engine branch Nov 19, 2016

@j0k3r j0k3r referenced this pull request Nov 19, 2016

Merged

wallabag 2.2.0 #2416

}
$qb
->andWhere('e.content LIKE :term OR e.title LIKE :term')->setParameter('term', '%'.$term.'%')

This comment has been minimized.

@damienalexandre

damienalexandre Nov 19, 2016

As you have Doctrine Extension already installed on the project, you could use https://github.com/beberlei/DoctrineExtensions/blob/master/src/Query/Mysql/MatchAgainst.php to do "real" token based full text search directly with MySQL 😋
It has the advantage of allowing to sort the result by relevance.

$qb->addSelect('match_against(p.titleFR, p.bodyFR) against (:search boolean ) AS _score');
$qb->having('_score > 0');
$qb->addOrderBy('_score', 'DESC');
$qb->setParameter('search', implode(',', $searchTokens));

I personally use this method to compute search tokens from user search:

    private function getSearchTokens($search)
    {
        $search = str_replace('  ', '', $search);
        $search = str_replace(' ', ',', $search);
        $search = str_replace('-', ',', $search);
        $search = str_replace('+', '', $search);
        $search = Transliterator::urlize($search, ',');

        $tokens = array_filter(explode(',', $search));

        // Add "mandatory" prefix
        array_walk($tokens, function(&$item) { $item = '+'.$item; });

        return $tokens;
    }
@damienalexandre

damienalexandre Nov 19, 2016

As you have Doctrine Extension already installed on the project, you could use https://github.com/beberlei/DoctrineExtensions/blob/master/src/Query/Mysql/MatchAgainst.php to do "real" token based full text search directly with MySQL 😋
It has the advantage of allowing to sort the result by relevance.

$qb->addSelect('match_against(p.titleFR, p.bodyFR) against (:search boolean ) AS _score');
$qb->having('_score > 0');
$qb->addOrderBy('_score', 'DESC');
$qb->setParameter('search', implode(',', $searchTokens));

I personally use this method to compute search tokens from user search:

    private function getSearchTokens($search)
    {
        $search = str_replace('  ', '', $search);
        $search = str_replace(' ', ',', $search);
        $search = str_replace('-', ',', $search);
        $search = str_replace('+', '', $search);
        $search = Transliterator::urlize($search, ',');

        $tokens = array_filter(explode(',', $search));

        // Add "mandatory" prefix
        array_walk($tokens, function(&$item) { $item = '+'.$item; });

        return $tokens;
    }

This comment has been minimized.

@nicosomb

nicosomb Nov 19, 2016

Member

Thank you @damienalexandre for your review 👍

to do "real" token based full text search directly with MySQL

We also support Postgres and SQLite. I didn't find MatchAgainst in SQLite and Postgres folders in DoctrineExtensions.

@nicosomb

nicosomb Nov 19, 2016

Member

Thank you @damienalexandre for your review 👍

to do "real" token based full text search directly with MySQL

We also support Postgres and SQLite. I didn't find MatchAgainst in SQLite and Postgres folders in DoctrineExtensions.

This comment has been minimized.

@damienalexandre

damienalexandre Nov 19, 2016

Damned, didn't know that :) I know Postgres has Full Text search capabilities but for SQLite it's a no-go. The Elasticsearch implementation will be needed then ^^
🍻 cheers 💛

@damienalexandre

damienalexandre Nov 19, 2016

Damned, didn't know that :) I know Postgres has Full Text search capabilities but for SQLite it's a no-go. The Elasticsearch implementation will be needed then ^^
🍻 cheers 💛

This comment has been minimized.

@nicosomb

nicosomb Nov 19, 2016

Member

I'm waiting your PR ;-)

@nicosomb

nicosomb Nov 19, 2016

Member

I'm waiting your PR ;-)

This was referenced Nov 20, 2016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment