New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add documentation search #37

Closed
wants to merge 1 commit into
base: master
from

Conversation

Projects
None yet
5 participants
@maxiloc
Contributor

maxiloc commented May 27, 2015

Context

I always wanted to have search capabilities inside the Laravel documentation so I made it.

PR Demo

You can test the live demo here http://laravel.neptos.fr/docs/5.0

This pull request is based on the pull request from @mattstauffer. It improves relevance, UX and adds instant search capabilities to the documentation using the Algolia API.

You will find a list of differences between the two pull requests at the end of this comment.

Benefits for the end user

  • Documentation search ALA Devdocs.
  • Relevance and speed thanks to Algolia search service (your search index is replicated in 12 regions worldwide, it is fast no matter where the user is)
  • Instant access to the documentation content (pages, sections, ...) at the first keystroke, in a few milliseconds

How does it works

There's a free index at Algolia that hosts the documentation, when a user does a search we use the Algolia API and then map to the pages and section of the documentation.

The index is populated via an Artisan task (more on this below).

Testing the pull request locally

If you want to test this pull request on your machine, you can do this:

git clone git@github.com:maxiloc/laravel.com.git
cd laravel.com
git checkout add-documentation-search
composer update
php artisan serve

Note:
This relies on your resources/docs/* Markdown folders being available locally.

Updating the index

When doing documentation updates, to update the relevant Algolia index, do this:

ALGOLIA_ADMIN_KEY=[YOUR_ALGOLIA_ADMIN_KEY] php artisan docs:index

@taylorotwell I am sending you a email with access to the Algolia's dashboard of the index.
You will then be able to get the ALGOLIA_ADMIN_KEY that will allow you to update the index (in the credentials dashboard).
This is your account, you are free to do whatever you want.

Differences with ElasticSearch pull-request (#18)

Relevance

I cut each page in small parts (h1, h2, ...) and prioritize them in order to have good relevance. It also allows to go directly to the concerning anchor when selecting a suggestion.

UI/UX

This is an autocomplete implementation, which allow a quick access to content using only keyboard and do not need a page load to have the results.

Hosting

The elastic search solution requires us to host it and maintain it, while Algolia is a SAAS hosted it in 12 different regions.

Algolia

We already support other community websites like Hackernews search, CDNJS, GrowthHackers or Product hunt. In exchange, we only ask for a "powered by Algolia" in the search results. This pull request adds it too.

If you want to move forward with this implementation, we will of course give you the search account for free as explained before.

Please contact me if you have any question or would like more information.

@taylorotwell

This comment has been minimized.

Show comment
Hide comment
@taylorotwell

taylorotwell Jun 2, 2015

Member

So I've got this working locally, and it is very nice. However, I am quite concerned about maintaining this custom block parsing logic going forward. If there is ever any change with our documentation parsing or library or logic, etc. everything related to search would break, which gives me some pause.

Can you give me a better idea of what all that is trying to do? Maybe it can be done in a better way?

Member

taylorotwell commented Jun 2, 2015

So I've got this working locally, and it is very nice. However, I am quite concerned about maintaining this custom block parsing logic going forward. If there is ever any change with our documentation parsing or library or logic, etc. everything related to search would break, which gives me some pause.

Can you give me a better idea of what all that is trying to do? Maybe it can be done in a better way?

@maxiloc

This comment has been minimized.

Show comment
Hide comment
@maxiloc

maxiloc Jun 2, 2015

Contributor

Hi, thanks for your response.

The main idea for this search was to have the best relevance possible. By cutting the markdown into small pieces, we can then prioritize some blocks like h1, h2 in the search ranking.

This is a different approach to indexing the all markdown/html directly. It generally leads to better relevance.

To do that there is two functions dealing with markdown Blocks :

The \App\CustomParser::getBlocks function

It is a copy Parsedown::lines without the end of the code that convert the $Blocks variable to html. I did not modify anything except cutting the end.

The original function can be found here https://github.com/erusev/parsedown/blob/790066e9a7dbac9b1441005b902531ec9de0ed00/Parsedown.php#L124 (I also added this as a comment in the code)

The App\Services\Documentation\Indexer::indexDocument function

It take the markdown blocs from \App\CustomParser::getBlocks, filter it by ignoring unwanted block like code example and for each one create an object that look like

[
        "objectID"           => "master-eloquent-relationships#inserting-related-models-4406844604fcfd5b7a9485ae0efefdb7",
        "h1"                 => "Eloquent: Relationships",
        "h2"                 => "Inserting Related Models"
        "h3"                 => null
        "h4"                 => null
        "link"               => eloquent-relationships#inserting-related-models"
        "content"            => null
        "importance"         => 1
        "_tags"              => ["master"]
]

If the documentation moves from markdown to another format then yes we will maybe have to do some work.

In all cases, I will always be available to help you on this feature.

Let me know what you think.

Contributor

maxiloc commented Jun 2, 2015

Hi, thanks for your response.

The main idea for this search was to have the best relevance possible. By cutting the markdown into small pieces, we can then prioritize some blocks like h1, h2 in the search ranking.

This is a different approach to indexing the all markdown/html directly. It generally leads to better relevance.

To do that there is two functions dealing with markdown Blocks :

The \App\CustomParser::getBlocks function

It is a copy Parsedown::lines without the end of the code that convert the $Blocks variable to html. I did not modify anything except cutting the end.

The original function can be found here https://github.com/erusev/parsedown/blob/790066e9a7dbac9b1441005b902531ec9de0ed00/Parsedown.php#L124 (I also added this as a comment in the code)

The App\Services\Documentation\Indexer::indexDocument function

It take the markdown blocs from \App\CustomParser::getBlocks, filter it by ignoring unwanted block like code example and for each one create an object that look like

[
        "objectID"           => "master-eloquent-relationships#inserting-related-models-4406844604fcfd5b7a9485ae0efefdb7",
        "h1"                 => "Eloquent: Relationships",
        "h2"                 => "Inserting Related Models"
        "h3"                 => null
        "h4"                 => null
        "link"               => eloquent-relationships#inserting-related-models"
        "content"            => null
        "importance"         => 1
        "_tags"              => ["master"]
]

If the documentation moves from markdown to another format then yes we will maybe have to do some work.

In all cases, I will always be available to help you on this feature.

Let me know what you think.

@maxiloc

This comment has been minimized.

Show comment
Hide comment
@maxiloc

maxiloc Jun 3, 2015

Contributor

@taylorotwell I rebased on your last commits. The PR is ready to be merged!!! 😄

Contributor

maxiloc commented Jun 3, 2015

@taylorotwell I rebased on your last commits. The PR is ready to be merged!!! 😄

@MartelliEnrico

View changes

Show outdated Hide outdated config/app.php
Add documentation search
Context: I always wanted to have search capabilities inside the Laravel
documentation so I made it.

This pull request adds instant search capabilities to the documentation
using the Algolia API. I designed it to provide a good relevance via an
autocomplete UI.
@maxiloc

This comment has been minimized.

Show comment
Hide comment
@maxiloc

maxiloc Jun 6, 2015

Contributor

@MartelliEnrico Thanks. Done

Contributor

maxiloc commented Jun 6, 2015

@MartelliEnrico Thanks. Done

@peterpan666

This comment has been minimized.

Show comment
Hide comment
@peterpan666

peterpan666 Jun 8, 2015

Nice work on that!

peterpan666 commented Jun 8, 2015

Nice work on that!

{
$indexer->indexAllDocuments();
}
}

This comment has been minimized.

@GrahamCampbell
@GrahamCampbell
{
$bus->dispatch(new IndexAllDocuments);
}
}

This comment has been minimized.

@GrahamCampbell
@GrahamCampbell
@@ -26,4 +27,4 @@ protected function schedule(Schedule $schedule)
->hourly();
}
}
}

This comment has been minimized.

@GrahamCampbell
@GrahamCampbell
class CustomParser extends \ParsedownExtra {
/**
* @param $markdown

This comment has been minimized.

@GrahamCampbell

GrahamCampbell Jun 8, 2015

Member

You need the correct phpdoc here.

@GrahamCampbell

GrahamCampbell Jun 8, 2015

Member

You need the correct phpdoc here.

* @return array
*
* Partial copy of the lines function from \Parsdown Class to get the Blocks
* https://github.com/erusev/parsedown/blob/790066e9a7dbac9b1441005b902531ec9de0ed00/Parsedown.php#L124

This comment has been minimized.

@GrahamCampbell
@GrahamCampbell
return $Blocks;
}
}

This comment has been minimized.

@GrahamCampbell
@GrahamCampbell
@taylorotwell

This comment has been minimized.

Show comment
Hide comment
@taylorotwell

taylorotwell Jun 8, 2015

Member

I've been working on all that Graham.

Member

taylorotwell commented Jun 8, 2015

I've been working on all that Graham.

@taylorotwell

This comment has been minimized.

Show comment
Hide comment
@taylorotwell

taylorotwell Jun 8, 2015

Member

No need for the OP to make anymore changes.

Member

taylorotwell commented Jun 8, 2015

No need for the OP to make anymore changes.

return str_replace('.md', '', $fileName);
}
}

This comment has been minimized.

@GrahamCampbell
@GrahamCampbell
@@ -8,7 +8,8 @@
"laravel/framework": "5.0.*",
"league/commonmark": "0.7.*",
"erusev/parsedown-extra": "0.7.*",
"symfony/browser-kit": "~2.3"
"symfony/browser-kit": "~2.3",
"vinkla/algolia": "1.1.0"

This comment has been minimized.

@GrahamCampbell

GrahamCampbell Jun 8, 2015

Member

This shouldn't be locked at that version. use 1.1.* or ~1.1.

@GrahamCampbell

GrahamCampbell Jun 8, 2015

Member

This shouldn't be locked at that version. use 1.1.* or ~1.1.

@mattstauffer mattstauffer referenced this pull request Jun 8, 2015

Closed

L5 searchable docs #18

@maxiloc

This comment has been minimized.

Show comment
Hide comment
@maxiloc

maxiloc Jun 8, 2015

Contributor

@taylorotwell I am very exited about the release tomorrow 😄.

I just created you a second index called "docs_lumen" if you want the same search on the Lumen documentation.

You will just need to replace in App\Services\Documention\Indexer

private static $index_name = 'docs';

by

private static $index_name = 'docs_lumen';

And change in laravel.js

var index = client.initIndex('docs');

by

var index = client.initIndex('docs_lumen');
Contributor

maxiloc commented Jun 8, 2015

@taylorotwell I am very exited about the release tomorrow 😄.

I just created you a second index called "docs_lumen" if you want the same search on the Lumen documentation.

You will just need to replace in App\Services\Documention\Indexer

private static $index_name = 'docs';

by

private static $index_name = 'docs_lumen';

And change in laravel.js

var index = client.initIndex('docs');

by

var index = client.initIndex('docs_lumen');
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment