Include a searcher for classes/methods/properties #636

Open
dcastanos opened this Issue Oct 9, 2012 · 21 comments

Comments

Projects
None yet
5 participants
@dcastanos

It would be nice to have a searcher (with autocomplete) to quickly find classes/methods/properties.

@mvriel

This comment has been minimized.

Show comment
Hide comment
@mvriel

mvriel Oct 13, 2012

Member

I absolutely agree; we have had a search option until a few months back when we noticed that some of the larger projects became unusable due to the large index and load times induced by search.

This is however an issue that needs to be solved; however I haven't had time to properly address this. Basically we need to write our own search engine in JS that performs well

Member

mvriel commented Oct 13, 2012

I absolutely agree; we have had a search option until a few months back when we noticed that some of the larger projects became unusable due to the large index and load times induced by search.

This is however an issue that needs to be solved; however I haven't had time to properly address this. Basically we need to write our own search engine in JS that performs well

@dcastanos

This comment has been minimized.

Show comment
Hide comment
@dcastanos

dcastanos Oct 15, 2012

Maybe you can take a look to the ApiGen searcher code... :-P

Maybe you can take a look to the ApiGen searcher code... :-P

@mvriel

This comment has been minimized.

Show comment
Hide comment
@mvriel

mvriel Oct 15, 2012

Member

I could, but then I'd have the same issue since they haven't solved it as well. :)

Member

mvriel commented Oct 15, 2012

I could, but then I'd have the same issue since they haven't solved it as well. :)

@dcastanos

This comment has been minimized.

Show comment
Hide comment

Good one!

@boenrobot

This comment has been minimized.

Show comment
Hide comment
@boenrobot

boenrobot Oct 15, 2012

Contributor

Is there a possibility of maybe having an option to turn on the old searcher?

I can understand if it's off by default, and come with a big warning in the help, but still let people enable it at their own... errr... let's say "risk".

If the problem is ever solved, it could then be turned on by default.

Contributor

boenrobot commented Oct 15, 2012

Is there a possibility of maybe having an option to turn on the old searcher?

I can understand if it's off by default, and come with a big warning in the help, but still let people enable it at their own... errr... let's say "risk".

If the problem is ever solved, it could then be turned on by default.

@mvriel

This comment has been minimized.

Show comment
Hide comment
@mvriel

mvriel Oct 15, 2012

Member

Enabling the searcher is a matter of adding the 'Search' writer to your template.xml and providing an input that can read the generated search index, such as this: https://github.com/phpDocumentor/template.new_black/blob/master/search/ajax.xsl
(though I honestly do not know for sure whether that last bit still works)

Member

mvriel commented Oct 15, 2012

Enabling the searcher is a matter of adding the 'Search' writer to your template.xml and providing an input that can read the generated search index, such as this: https://github.com/phpDocumentor/template.new_black/blob/master/search/ajax.xsl
(though I honestly do not know for sure whether that last bit still works)

@mvriel

This comment has been minimized.

Show comment
Hide comment
@mvriel

mvriel Oct 15, 2012

Member

You can make that configurable, but that would mean setting a parameter on the template.xml or using the config file and altering the template + search writer to query that parameter

Member

mvriel commented Oct 15, 2012

You can make that configurable, but that would mean setting a parameter on the template.xml or using the config file and altering the template + search writer to query that parameter

@boenrobot

This comment has been minimized.

Show comment
Hide comment
@boenrobot

boenrobot Oct 15, 2012

Contributor

Personally, I'd prefer the second option (config file option + the template and writer querying that option). In the template's XSLT file, this option's value should be accessible from an XSLT parameter.

Contributor

boenrobot commented Oct 15, 2012

Personally, I'd prefer the second option (config file option + the template and writer querying that option). In the template's XSLT file, this option's value should be accessible from an XSLT parameter.

@mvriel

This comment has been minimized.

Show comment
Hide comment
@mvriel

mvriel Dec 22, 2012

Member

Work has started on a search implementation in https://github.com/mvriel/phpDocumentor2/tree/search
This implementation focuses on ElasticSearch as a backend but the design uses adapters to be able to use other backends

Member

mvriel commented Dec 22, 2012

Work has started on a search implementation in https://github.com/mvriel/phpDocumentor2/tree/search
This implementation focuses on ElasticSearch as a backend but the design uses adapters to be able to use other backends

@dcastanos

This comment has been minimized.

Show comment
Hide comment
@dcastanos

dcastanos Apr 5, 2013

Which is the status of this development?

Thank you very much.

Which is the status of this development?

Thank you very much.

@mvriel

This comment has been minimized.

Show comment
Hide comment
@mvriel

mvriel Apr 5, 2013

Member

Currently I am working on a monster refactoring (#792) before 2.0 goes beta. After that is done is this issue one of the first to be picked up for 2.1

Member

mvriel commented Apr 5, 2013

Currently I am working on a monster refactoring (#792) before 2.0 goes beta. After that is done is this issue one of the first to be picked up for 2.1

@dcastanos

This comment has been minimized.

Show comment
Hide comment
@dcastanos

dcastanos Apr 5, 2013

Nice!

Thank you very much

Nice!

Thank you very much

mvriel added a commit to mvriel/phpDocumentor2 that referenced this issue Sep 6, 2013

@dcastanos

This comment has been minimized.

Show comment
Hide comment
@dcastanos

dcastanos Oct 3, 2013

Hi!
Version 2.1 is out, but the searcher isn't in yet, is it?

Hi!
Version 2.1 is out, but the searcher isn't in yet, is it?

@mvriel

This comment has been minimized.

Show comment
Hide comment
@mvriel

mvriel Oct 7, 2013

Member

Version 2.1 is not yet out, it is delayed because I did not have time to finish the release. Next weekend I will release it. It looks like the searcher will not be in due to time constraints; work on it will continue in the next sprint

Member

mvriel commented Oct 7, 2013

Version 2.1 is not yet out, it is delayed because I did not have time to finish the release. Next weekend I will release it. It looks like the searcher will not be in due to time constraints; work on it will continue in the next sprint

@ghost ghost assigned mvriel Oct 14, 2013

@mvriel

This comment has been minimized.

Show comment
Hide comment
@mvriel

mvriel Dec 5, 2013

Member

Unfortunately due to real life I have had to postpone work o the search implementation (life is what happens when you are busy making plans); this item is not forgotten but replanned for 2.3

Member

mvriel commented Dec 5, 2013

Unfortunately due to real life I have had to postpone work o the search implementation (life is what happens when you are busy making plans); this item is not forgotten but replanned for 2.3

@dcastanos

This comment has been minimized.

Show comment
Hide comment
@dcastanos

dcastanos Dec 9, 2013

Ok, thanks for the update.

Ok, thanks for the update.

@mvriel mvriel modified the milestones: 2.4, 2.3 Feb 16, 2014

@dcastanos

This comment has been minimized.

Show comment
Hide comment
@dcastanos

dcastanos Feb 20, 2014

HI again mvriel... 2.3 is out, any news?

HI again mvriel... 2.3 is out, any news?

mvriel added a commit to mvriel/phpDocumentor2 that referenced this issue Mar 2, 2014

mvriel added a commit to mvriel/phpDocumentor2 that referenced this issue Mar 2, 2014

#636: Added EngineManager and refactored code
In this commit I cleaned up the code by adding DocBlocks and renaming
the Engine namespace to Adapter in order to be more explicit in our
intention with that code.

In order to provide a clear entry point and extension point I added an
EngineManager class, inspired on Doctrine's EntityManager, that calls
the correct adapter and works as a frontend.

mvriel added a commit to mvriel/phpDocumentor2 that referenced this issue Mar 2, 2014

mvriel added a commit to mvriel/phpDocumentor2 that referenced this issue Mar 2, 2014

mvriel added a commit to mvriel/phpDocumentor2 that referenced this issue Mar 2, 2014

#636: Refactored Client Code Generator and added test
The client code generator did not support creating multiple types of
clients. Since most search engines will have a Javascript search option
and a PHP (or other) server side script that communicates with that
Search Engine the generator should be able to create frontend and
backend code.

mvriel added a commit to mvriel/phpDocumentor2 that referenced this issue Mar 2, 2014

#636: Fix broken @cover tags
When the tests were moved from their old location to a plugin the
namespaces in the @cover tags did not get updated. This commit fixes
those broken namespaces.

mvriel added a commit to mvriel/phpDocumentor2 that referenced this issue Mar 2, 2014

#636: Add documentation for Document
The document class did not feature any documentation. This commit adds
the necessary DocBlocks to the entity.

mvriel added a commit to mvriel/phpDocumentor2 that referenced this issue Mar 2, 2014

#636: EngineManager should return Adapter
In order to generate client code the adapter is used to determine the
type of client to generate. To make this possible the engine manager
now returns the adapter and a test is added to verify that.

mvriel added a commit to mvriel/phpDocumentor2 that referenced this issue Mar 2, 2014

#636: Added Search writer
The Search writer populates the Search Engine and writes code for the
frontend and backend side of a client to interact with the selected
Search Engine.

mvriel added a commit to mvriel/phpDocumentor2 that referenced this issue Mar 2, 2014

#636: Extract interface from generator
Because the generator can be replaced in the container it is a good
thing to extract an interface from the concrete Generator and use that
for typehinting.

mvriel added a commit to mvriel/phpDocumentor2 that referenced this issue Mar 2, 2014

#636: Extract Generator
The Generator class is instantiated inside the Search writer. To ease
testing and remove the dependency with Twig we move the Generator
instantiation to the container and add it as a dependency.

mvriel added a commit to mvriel/phpDocumentor2 that referenced this issue Mar 2, 2014

mvriel added a commit to mvriel/phpDocumentor2 that referenced this issue Mar 2, 2014

#636: Update ServiceProvider
The ServiceProvider does not instantiate all the necessary dependencies
and contained errors after a previous refactoring. In this commit we
update the service provider to include all changes up til now.

mvriel added a commit to mvriel/phpDocumentor2 that referenced this issue Mar 2, 2014

#636: Removed mapper
The mapper was initially meant to map fields in a descriptor to the
Document class. This behaviour was so minute that we implemented it in
the writer class as a method; perhaps if it proves to be worthwhile
later we can reintroduce this concept.

mvriel added a commit to mvriel/phpDocumentor2 that referenced this issue Mar 2, 2014

#636: Extract Configuration to Interface
In order to improve type-hinting and reduce things like duck-typing
I have implemented an interface above the configuration.

mvriel added a commit to mvriel/phpDocumentor2 that referenced this issue Mar 2, 2014

#636: Add tests for the LunrJs Configuration class.
In addition to adding tests the configuration class was also cleaned up
and documented.

mvriel added a commit to mvriel/phpDocumentor2 that referenced this issue Mar 2, 2014

#636: Add tests for the ElasticSearch Configuration class.
In addition to adding tests the configuration class was also cleaned up
and documented.
@mvriel

This comment has been minimized.

Show comment
Hide comment
@mvriel

mvriel Mar 17, 2014

Member

Hi @dcastanos, it takes more work than expected. I have pushed some changes so you can see the progress what I made. The work that is now required is to integrate the library that I have written into templates and a service provider to that we can start testing it out

Member

mvriel commented Mar 17, 2014

Hi @dcastanos, it takes more work than expected. I have pushed some changes so you can see the progress what I made. The work that is now required is to integrate the library that I have written into templates and a service provider to that we can start testing it out

@dcastanos

This comment has been minimized.

Show comment
Hide comment
@dcastanos

dcastanos Mar 28, 2014

Hi mvriel! Great news!

Thanks a lot for the update.

Hi mvriel! Great news!

Thanks a lot for the update.

@mvriel mvriel modified the milestones: 2.5, 2.4 Apr 1, 2014

mvriel added a commit to mvriel/phpDocumentor2 that referenced this issue Apr 26, 2014

mvriel added a commit to mvriel/phpDocumentor2 that referenced this issue Apr 26, 2014

@mvriel mvriel modified the milestones: 2.6, 2.5, 2.7 May 17, 2014

@mvriel mvriel modified the milestones: 2.7, 2.8 Aug 30, 2014

@mvriel mvriel modified the milestones: Blue Lightning, 2.8 Oct 23, 2014

@mvriel mvriel added the Roadmap label Oct 23, 2014

@mvriel mvriel modified the milestones: Blue Lightning, 3.0 Feb 18, 2015

@mvriel mvriel removed the Improvement label Aug 22, 2015

@mvriel mvriel removed their assignment Aug 22, 2015

@ghost ghost referenced this issue in userfrosting/UserFrosting Sep 17, 2015

Closed

documentation / docs #272

@mvriel mvriel added the in progress label Dec 28, 2015

@mvriel mvriel self-assigned this Dec 28, 2015

@mvriel

This comment has been minimized.

Show comment
Hide comment
@mvriel

mvriel Dec 28, 2015

Member

This feature is planned to be included into phpDocumentor 3.0.

I have been reviewing several options including lunr.js and elasticlunr.js but the biggest issue so far is that client-side building of the FTS index requires Node.js. This is a requirement that is hard to sell (imo) and building the index client side requires a large JSON element that needs to be included in every page load; making for terrible performance.

I am now reviewing https://github.com/cebe/js-search by @cebe to check whether that fulfills the requirements that we have for search. The advantage of this package is that the index is pre-rendered in PHP, the most interesting part here is performance on a codebase the size of Zend Framework or Symfony and how large the index becomes since it needs to be included on each page load. (AJAX loading cannot be supported since offline browsing doesn't do AJAX requests; though this may be circumvented using JSONP)

Member

mvriel commented Dec 28, 2015

This feature is planned to be included into phpDocumentor 3.0.

I have been reviewing several options including lunr.js and elasticlunr.js but the biggest issue so far is that client-side building of the FTS index requires Node.js. This is a requirement that is hard to sell (imo) and building the index client side requires a large JSON element that needs to be included in every page load; making for terrible performance.

I am now reviewing https://github.com/cebe/js-search by @cebe to check whether that fulfills the requirements that we have for search. The advantage of this package is that the index is pre-rendered in PHP, the most interesting part here is performance on a codebase the size of Zend Framework or Symfony and how large the index becomes since it needs to be included on each page load. (AJAX loading cannot be supported since offline browsing doesn't do AJAX requests; though this may be circumvented using JSONP)

@cebe

This comment has been minimized.

Show comment
Hide comment
@cebe

cebe Dec 29, 2015

Contributor

Thanks for mentioning me. Note that this thing is in experimental state and might need some improvement. It currently also does not work on the basis of classes or property/method names but does some full text search things on the HTML which has simply stripped the tags.

The current index size for Yii framework docs is about 450KB (gzipped) http://www.yiiframework.com/doc-2.0/jssearch.index.js

When you are only interested in searching classes or methods the size of the index can be reduced much more. There is some code I wrote for the rewrite of the yiiframework.com website which we are currently working on: https://github.com/yiisoft-contrib/yiiframework.com/blob/837e4ae8d24c02283cca45158ae68f087f39b35d/js/search.js

This uses an index of classes: http://yiiframework.domain-na.me/doc/api?_format=json
And one for method and property names: http://yiiframework.domain-na.me/doc/api/class-members?_format=json

You can see it in action on http://yiiframework.domain-na.me/ which shows the current state of site development. If you type into the search box, it will show you class names. When you type a . it will let you search for methods or property names, e.g.:

ActiveRecord  # shows class results
.open  # shows methods or properties
::open  # same as above
Active.save  # finds methods that contain save in classes that contain Active

Also note that if you load the search index as JS files, they will be loaded on first use and afterwards are stored by browser cache until they expire.

Contributor

cebe commented Dec 29, 2015

Thanks for mentioning me. Note that this thing is in experimental state and might need some improvement. It currently also does not work on the basis of classes or property/method names but does some full text search things on the HTML which has simply stripped the tags.

The current index size for Yii framework docs is about 450KB (gzipped) http://www.yiiframework.com/doc-2.0/jssearch.index.js

When you are only interested in searching classes or methods the size of the index can be reduced much more. There is some code I wrote for the rewrite of the yiiframework.com website which we are currently working on: https://github.com/yiisoft-contrib/yiiframework.com/blob/837e4ae8d24c02283cca45158ae68f087f39b35d/js/search.js

This uses an index of classes: http://yiiframework.domain-na.me/doc/api?_format=json
And one for method and property names: http://yiiframework.domain-na.me/doc/api/class-members?_format=json

You can see it in action on http://yiiframework.domain-na.me/ which shows the current state of site development. If you type into the search box, it will show you class names. When you type a . it will let you search for methods or property names, e.g.:

ActiveRecord  # shows class results
.open  # shows methods or properties
::open  # same as above
Active.save  # finds methods that contain save in classes that contain Active

Also note that if you load the search index as JS files, they will be loaded on first use and afterwards are stored by browser cache until they expire.

@mvriel mvriel removed this from the 3.0 - Migrate to a DDD Architecture milestone May 19, 2017

@jaapio jaapio added the Template label Dec 25, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment