The format of stork.js is different from the example

[petia-latinova]

Hi,

I downloaded the project because I wanted to fix the issue when the search crashes on very common word like "and" when you have too many results. I think I have success. It works from command line but I want to test it in my web project before give you the fix because i still don't know how will looks like. There I also have some questions about some limitations that's why I am still not sure that the fix is ok.

My problem is that I can't build the stork.js file in the same format as https://files.stork-search.net/stork.js
Because of the specifics in my project I needed to have stork variable in window.stork which is easy to change in the file above but when I compile the project the stork.js file have different format. Maybe there is something in the packaging that change it? I'll really appreciate if you help me to build that file and hopefully if I manage to make successful fix I can share it.

Best Regards,
Petia

[petia-latinova]

I think that the problem with building the stork.js file is because I tried to build it under macos. If is build under linux the file format is the same so maybe we need to close that issue. Sorry about that.

Best Regards,
Petia

[jameslittle230]

Hey @petia-latinova -- yeah, that's probably it.

Feel free to open a PR! Even if you can't build the project, I'm happy to help get it merged in. I'm appreciative that you'd take the time to make Stork better! I'm moving out of my apartment this week, though, and probably can't be very hands-on to help you get set up, so a PR (even if it's a little broken) would be the best way for you to get things rolling.

Thanks again!
James

[petia-latinova]

Hi @jameslittle230,
I am not sure that there is need for PR. I managed to make some small changes successful. They work good for me and I think that looks ok but maybe you can find better decision.

The fix for that issue:
#158

was to cut some functionality from the code for the title. I am not sure you'll like that one so I'll just send you the line.

In the file:
/stork/src/index_versions/v3/search/entry_and_intermediate_excerpts.rs

I changed line 143:
let beginning = split_title[0..ie.word_index].join(" ").len() + space_offset;

with:
let beginning = space_offset;

And doesn't crashes anymore. :)

Stork is awesome! I am really very happy to see it integrated in my project!
I built big *st file to check it's limits.

If you don't mind I'll ask few questions. I am not sure that this issue is the best place but I already started here so I hope you don't mind.

Can you tell me why there is limit for Stork to search in max 10files? I removed the limitation because I want to be sure I search in all of the files and is working good in my project. I tried it with large file and doesn't look like causes some trouble, Stork is still working perfect.

The other question worries me a bit more. When I try Stork with very large data file when I open my project Stork of course needs more time to load the *st file. And just at the end of the waiting everything is freezing for a bit even sometimes my app crashes if I click at the same time with the mouse. That happen with really large file. First I thought the problem is because parsing the json in that file:
/stork/js/entity.ts
is synchronous so that blocks everything:

const indexInfo = JSON.parse(wasm_register_index(this.name, data));

But I made experiment and made the json.parse asynchronous. It was bloody slow of course and didn't fixed the problem. So something else is the reason for that happening. Is it because maybe Stork add all the elements hidden to the DOM tree for faster searching?

Best Regards,
Petia

[jameslittle230]

@petia-latinova - sorry for the delay. I'm not sure if you're still having trouble/working with Stork, but in case you are, I'd like to answer your questions:

Why is there a limit on the files Stork can find?

You may have found this, but you can increase the limit from ten files by changing the displayed_results_count configuration setting (scroll all the way down to the bottom of the Configuration Reference to read more).

There's a default limit of 10 because increasing this value results in larger indexes, and this seemed like a fair balance. You're more than welcome to increase it if it works for you.

The web page gets slow when searching through really large indexes, even when the search function is run async

This is a limitation of Javascript's threading model. Browsers only really have one thread to run Javascript, and doing too much work on that thread will slow down other Javascript operations (to learn more about this, you can read about the event loop's "run-to-completion" behavior).

You don't usually notice this limitation with asynchronous functions because most asynchronous work is not made asynchronous because two heavy functions need to run concurrently; instead, they are asynchronous because they're waiting for something (a timer, a network request, a user event). Even if search were marked as asynchronous, the processing still takes place in the browser's main thread, so all other JS operations have to wait for that message to be completed before starting.

Eventually, I want the Stork JS library to run Stork searches on a Web Worker, which provides a true concurrency model. This would let searches truly take place in a "background process" and would fix this janky behavior.

[jameslittle230]

@petia-latinova - I'm going to close this issue because I don't think it represents a bug report or feature request. If you have a bug report or feature request, please feel free to reopen this issue or file a new one. If you want to chat more about the project, please feel free to open a Github Discussion!

Best,
James