Add Homemade Full-Text Search to Examples Site

[andygroff]

I was struggling to find things I need in the site so I attempted to upgrade the search functionality to a pseudo full text search algorithm.

To accomplish this, it first hooks into the npm run update command and generates an index json file at public/documentIndex.json. On the front-end, the current search implementation was replaced with a pop-up style search interface which loads the index data and then attempts to score each example against the query using a simple algorithm I came up with via trial and error.

The only issue I'm aware of this causing is it seems to be linking to files that are not available through the browse interface. In order to fix that we'll need to update the exclude regular expression in the node.js file which produces the index. I wasn't totally sure of that logic and figured the examples could still be useful so I haven't fixed it.

Example Query:

[photonstorm]

Very nice :) The logic on what to exclude is simple for the most part:

1. If the folder name is called archive then skip it
2. If the folder name starts with an underscore, skip all contents
3. If a filename starts with an underscore, skip it

That would exclude the majority of it. The only other files it would pick-up that it shouldn't would be where you've got a boot.json in a folder along with lots of source files, in those cases the boot is the 'runner' that loads in all the other source files, requireJS style.

However, I don't think it'll matter too much including them in the results. I could update the node build script to add in an extra property ignore that the front-end and search could use.

[andygroff]

Thanks for the clarification. I have updated the indexing process to exclude things with the given logic. I think it is all working as expected. I also removed the file extensions from the UI and put in a special condition so it handles boot.json properly.

[photonstorm]

Thanks - this is merged so it'll auto-deploy to the main site shortly.

[andygroff]

I just saw! Thanks for merging it, I am super excited. Feel free to ping me if there's any issues or questions with anything. Thanks

[photonstorm]

I notice when I do node build it's not picking up the new files I added locally, and the files are no longer in alphabetical order when viewing them. Before I dig in to fix this, is it because of the changes to the build script, or the addition of the search indexer?

[photonstorm]

Actually, it's picking up the new files fine, was a Safari cache issue. They're not in alphabetical order though, which I'm wondering might be because I ran the script on my new Mac (I usually use Windows) and that may be the culprit.

[andygroff]

Hmmm well there's definitely nothing I did intentionally to change the sort order of the filteredTree variable. It should be just recursing through the tree and creating a new flat array from it, but I know javascript does lots of passing by reference so it's possible I am doing an operation somewhere that changes that variable. A simple test would just comment out the indexing process completely on line ten of build.js. Alternatively, I can pass a deep copy of that variable into the indexer so that there's no chance of any side effects to the pre-existing build process.

However it does seem that the culprit is more likely to be some combination of nodejs and operating system issues. I'm also on mac using node v8.9.4 and everything is sorted properly for me. The directory-tree module you're using generates the children list with node's fs.readDirSync method which does seem to have some version/platform related issues with inconsistent ordering and doesn't mention any guaranteed order.

Since the user interface does depend on the ordering being alphabetical and directory-tree doesn't guarantee any order or sorting I think the best approach would be to manually sort that tree alphabetically to ensure consistency across platforms and node versions. I could implement tonight after work if you'd like.