Skip to content

Replace tipue search with lunr.js and lunr.py #275

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
codecachet opened this issue Feb 7, 2019 · 22 comments
Closed

Replace tipue search with lunr.js and lunr.py #275

codecachet opened this issue Feb 7, 2019 · 22 comments
Assignees
@talha131
Labels
enhancement pull request welcome Not a priority right now, but if you provide a pull request, we would merge it. released
Milestone
Roadmap

Comments

@codecachet
Copy link

I am distinctly getting the impression that tipue search is no longer being maintained, or is otherwise less than desirable. I suggest that we replace tipue search with lunr - lunr.py to create the index, and lunr.js to query the index when a search is launched. Both projects seem alive and well, and both Jekyll and Hugo seem to recommend using lunr for search.

Just a thought - like others, I have been very frustrated trying to get search to work for Elegant. One thing about tipue is I am not crazy about the jQuery dependency. I know jQuery is a dependency of Elegant currently, but if we get rid of Bootstrap, we could get rid of jQuery.
@talha131 talha131 added this to the Roadmap milestone Feb 10, 2019
@talha131
Copy link
Member

I too want to get rid of jquery. It's on our roadmap.

As for tipue. When elegant was first released, tipue was the only available search solution for static sites. Now we have much better and powerful options like

  1. https://fusejs.io/
  2. https://github.com/fergiemcdowall/search-index
  3. https://lunrjs.com/
  4. https://github.com/nextapps-de/flexsearch

Down the line, we should replace or add support for other search solutions too.

@iranzo
Copy link
Member

iranzo commented Jun 9, 2019

@codecachet do you want to create a PR for this?

@iranzo iranzo added enhancement pull request welcome Not a priority right now, but if you provide a pull request, we would merge it. labels Jun 9, 2019
@talha131 talha131 self-assigned this Jun 9, 2019
@talha131
Copy link
Member

talha131 commented Jun 9, 2019

@iranzo this is a big change. It should be handled by the core team. More importantly, we need to compare different solutions first.

@talha131
Copy link
Member

talha131 commented Jul 5, 2019

An issue noticed with tipue is that it treats plural differently from singular

Search for "variable" and then "variables" in the docs site. Result is different.

We can use stem words option to fix it but then how many words are we going to add in the list. Tipue should have this feature by default.

@talha131 talha131 mentioned this issue Jul 5, 2019
Closed
@iranzo
Copy link
Member

iranzo commented Jan 28, 2020

I'm looking at lunr.js for other aspects, seems that the data it uses for the search can be contained/generated on publish time without plugin, just a jinja loop (at least on jekyll). I'll provide some information once I finish my investigation.

@talha131
Copy link
Member

@iranzo I appreciate that you have taken up this task. Before you go ahead and look deeply into lunr.js, can you please step back and decide which library we should use.

In terms of Github stars, Fuse is way ahead of lunr.js, which is followed by flex search.

Github stars are not a reliable metric but they do give a pretty good idea about the community interest and support, which we are going to need.

Then there is performance factor. Flex claims that it is the most performant.

Finally, we have to consider the ease of use for the end user.

Can you please investigate these three libraries and let me know how they fare on the criteria I mentioned?

@iranzo
Copy link
Member

iranzo commented Jan 29, 2020

Hi, No worries, just wanted to mention that I was testing it as a website I'm using has it integrated (but it was pretty old version and using jekyll) about features. Lunr is not that bad, but my concern here is that it, as with tipue, pregenerates a javascript array with all website content... that means that for a big website, it will have to preload all information in the html generated and then perform the search.

I don't mind looking at flex directly to see how it performs, but probably doing perf test will require lot of articles and I was thinking that some one here in the team had a busy one that could be used for testing if the website source is available somewhere.

@talha131
Copy link
Member

This is the result of flex search benchmark that I got.

Flex is much more performant than other libraries. How well it is supported and how actively it is under development is to be investigated.

@talha131
Copy link
Member

that means that for a big website, it will have to preload all information in the html generated and then perform the search.

You hit the nail right on the head.

If we can avoid it using these new libraries then it would become a high priority feature to implement and ship.

@talha131
Copy link
Member

Flex search is fast but development has seem to have lost momentum.

  1. https://github.com/nextapps-de/flexsearch/graphs/commit-activity
  2. Status of the project? nextapps-de/flexsearch#150

Also, the documentation is not friendly. We will have a hard time making a head and tail of it.

@talha131
Copy link
Member

that means that for a big website, it will have to preload all information in the html generated and then perform the search.

I just tested lunrjs demo. Even if we have to ship a JSON object with all the contents of the site, just like TipueSearch, lunarjs has better search results.

Try searching for "assignment" in the demo here
It shows "assigned" in the results too, which is desirable.

Tipue cannot handle plural and singular words correctly.

I think lunrjs is a better option.

@talha131
Copy link
Member

Questions and work items for whoever pick up this task.

  • Do we need lunr.py because we already have a tipue search plugin, that can be forked to work with lunr.js
  • How do we show the search results? Tipue generates the search result page for us. With lunrjs, we will have to do it ourselves.

@iranzo
Copy link
Member

iranzo commented Jan 29, 2020

From what I've seen about lunr.js: The result is generated by jekyll when the page for 'search' is created, it contains array for all the text in the website.

Search page replaces text in the 'search results' area with the output of the script

Right now the tipue_search plugin does the generation of the search results as lunr just uses a backend for the index that needs to be populated it could also probably be done with current tipue_search plugin output or even try to avoid that plugin to get one 'easier' by no longer requiring ap plugin (if possible), as I've tested this with jekyll and not with pelican

@talha131
Copy link
Member

talha131 commented Jan 29, 2020 via email

@talha131
Copy link
Member

talha131 commented Jan 31, 2020 •
edited
Loading

How to generate search result from lunr.js

  1. https://codewithhugo.com/hugo-lunrjs-search-index/
  2. https://matthewdaly.co.uk/blog/2015/04/18/how-i-added-search-to-my-site-with-lunr-dot-js/
  3. https://learn.cloudcannon.com/jekyll/jekyll-search-using-lunr-js/

@talha131
Copy link
Member

talha131 commented Feb 2, 2020

The json generated by Tipue Search plugin can be used by lunrjs without any changes.

All we need is to generate the search results page.

If anyone is willing to step up to write the code for search results page, then please let us know. I will work with you to save your time and effort, if needed.

@talha131 talha131 mentioned this issue Feb 3, 2020
Closed
@iranzo iranzo mentioned this issue Feb 3, 2020
Closed
5 tasks
@iranzo
Copy link
Member

iranzo commented Feb 3, 2020

@talha131 review #573

Note, the code just 'works', sytling should be used on the results functions and probably remove the tipue search javascript from website (and maybe put the lunr one)

I count on you for the 'styling' , in the current form for the PR, it works as a drop-in replacement of tipue with same requirements (for plugin, etc)

@talha131
Copy link
Member

talha131 commented Feb 3, 2020

Love it. Will do the rest.

@iranzo
Copy link
Member

iranzo commented Feb 3, 2020

image

talha131 pushed a commit that referenced this issue Feb 3, 2020
@iranzo @talha131
feat(search): replace tipue_search with lunr.js
9d60af1 
Signed-off-by: Pablo Iranzo Gómez <Pablo.Iranzo@gmail.com>

Update #275
talha131 pushed a commit that referenced this issue Feb 3, 2020
@semantic-release-bot
chore(release): 5.2.0 [skip ci]
f57da89 
# [5.2.0](V5.1.0...V5.2.0) (2020-02-03)

### Bug Fixes

* regression introduced due to cache busting ([b953a38](b953a38))
* **archives:** at smaller screen layout does not make the best use of available space ([a289e9d](a289e9d))
* **syntax:** code blocks with line numbers do not have the correct border radius ([3a34a3e](3a34a3e))
* **syntax:** improve the font size and height of download button ([07595b3](07595b3))
* **typography:** fix font size and style of TOC ([2d597ab](2d597ab)), closes [#509](#509)
* Categories and Tags pages do not have space below the header ([d78f4e5](d78f4e5))
* remove invalid CSS rule ([368cb2c](368cb2c))
* use consistent color for border ([c85b7b8](c85b7b8))
* use darker color for the site name ([19f37ca](19f37ca))
* **article:** use loclate_date attribute instead of hardcoding the date format ([1da552c](1da552c)), closes [#552](#552)
* **bootstrap:** WIP. add Bootstrapv2.3.2 sources file to the project ([43ee8eb](43ee8eb)), closes [#429](#429)
* **seo:** regression introduced in PR [#556](#556) ([28bcb85](28bcb85)), closes [#505](#505)
* **typography:** about me and my projects heading is not on one line ([6a7989c](6a7989c))
* **typography:** font size of superscript number in categories and tags page ([d0a7162](d0a7162))
* **typography:** override Bootstrap base font ([f6a83a6](f6a83a6)), closes [#429](#429)
* **typography:** top menu bar does not have correct height ([d30ce82](d30ce82))
* **typography:** use consistent font and color for project list ([477a467](477a467))
* **typography:** use consistent font in the footer ([eeb0e30](eeb0e30))

### Features

* add support cache busting ([cde0dc5](cde0dc5))
* **categories:** change background color of uncollapsed Category ([6cc11b8](6cc11b8))
* **gallery:** add support for PhotoSwipe image gallery using raw HTML ([ad1bcea](ad1bcea)), closes [#567](#567)
* **js:** combine and minify all JS files into one to improve load speed ([f5047d4](f5047d4))
* **search:** improve search results page look ([73fa743](73fa743)), closes [#573](#573) [#275](#275)
* **search:** replace tipue_search with lunr.js ([9d60af1](9d60af1)), closes [#275](#275)
* **SEO:** Add header and documentation for Claiming Website on Yandex ([2bb691d](2bb691d))
* **SEO:** add SEO and SMO meta keywords in site's home page ([c5be0eb](c5be0eb))
* **sharing:** Enable customizing the sharing links ([f3e262f](f3e262f))
* **syntax:** add copy to clipboard button for every code snippet ([6f73317](6f73317)), closes [#574](#574) [#525](#525)
* **syntax:** add support for highlighting lines in code snippets ([ba2de8c](ba2de8c)), closes [#520](#520)
* **syntax:** add support for liquid_tags.include_code plugin ([0550a3a](0550a3a)), closes [#518](#518)
* **syntax:** use Gruvbox syntax highlighting theme ([0b50ab1](0b50ab1))
* **typography:** use consistent fonts ([8b7a8bd](8b7a8bd)), closes [#429](#429)
* **typography:** use consistent style for timestamps in tags, categories and archives pages ([a421dcd](a421dcd))
* **typography:** use dns-prefetch and preconnect to improve font load time ([e4bd582](e4bd582))
* **typography:** use responsive fonts ([39acdbf](39acdbf))
@talha131
Copy link
Member

talha131 commented Feb 3, 2020

🎉 This issue has been resolved in version 5.2.0 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀

@dclong
Copy link

dclong commented Dec 13, 2020

Does elegant still uses the tipue_search pelican plugin and only replaces the underlying tipue with lunr.js? Is configuration still the same?

@talha131
Copy link
Member

yeah. underlying configuration is same

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@talha131 talha131

Labels
enhancement pull request welcome Not a priority right now, but if you provide a pull request, we would merge it. released
Milestone
Roadmap
Development

No branches or pull requests
4 participants
@iranzo @dclong @talha131 @codecachet