Possible to use as a library?

[Ivshti]

Hello,

Obviously, this crate was designed to operate as a server, but there are a lot of cases where embedding it into an application make sense:

• using it in your own servers
• using it for a desktop application
• perhaps even compiling to WASM and using it on the front-end (e.g. with seed-rs)

There are nice reusable modules, such as the lexer, stopwords and the fst dependencies.

How feasible is it to split sonic into a library and a server that's based on that library?

I know about tantivy, but I think this crate would be great for this too

[valeriansaliou]

Hello. It's not on my ToDo, but I would not see what would make it impossible. I'm open to contributions!

[Ivshti]

Alright, thanks. Our team is considering adapting sonic if anything else doesn't work out. I just found this: https://www.reddit.com/r/rust/comments/bdj65n/simsearch_a_new_simple_and_lightweight_fuzzy/, might be useful for people with a similar problem

[valeriansaliou]

Note that simsearch does not provide file-system persistence. So you'd need to re-insert all your records if the application fails.

[Ivshti]

I forgot to mention that this is part of my goal.

What I'm looking for is a library that can, at application startup, build an in-memory search index from a vector of 20-30k items

[valeriansaliou]

Ah, okay! Not so sure Sonic is your fit then, as it stores everything on-disk as a persistent index.

[fabricedesre]

I'm interested in contributing to that, but before making changes I'd like to agree on what changes are needed.

The current server code makes heavy use of global variables using lazy_static in many places. This is problematic for a library, at least for some of these statics.

I would like to start by getting rid of APP_CONF. If we take StoreFSTPool as an example, various methods need access to APP_CONF.store.fst. We can either:

1. add a configuration parameter to each function that needs it.
2. add a config field to the StoreFSTPool struct, and pass an instance around.

@valeriansaliou what do you think?

[valeriansaliou]

Hello,

Sonic was never intended to be used as a library, what's your use case? Sonic could be split into a library and a server including that library in the future.

[fabricedesre]

I want to use it as an embedded FTS as part of a self-contained program. Similar to using Sqlite instead of MySql/PgSql.

[AlexMikhalev]

There is separate library for fst by burnedsushi. Will it cover your use case?

[fabricedesre]

There is separate library for fst by burnedsushi. Will it cover your use case?

No, the fst is just one building block, but as you see Sonic needs much more.

[fabricedesre]

@valeriansaliou any opinion on the approach in #150 (comment) ? I'd rather not maintain a fork because I think changes will propagate to a lot of places.

[AlexMikhalev]

I am interested in sonic as library as well - embedding into electron. I have looked into tantivy, but would like to progress with something more minimalist since I plan to build my own relevance.

[valeriansaliou]

@fabricedesre On your approach suggestion, I think we can keep a common configuration passed along to a single library initialization interface, the configuration would look a lot like current's Sonic server configuration, except w/o all the network binding part.

Also, you'd need a programmatic way to talk to Sonic the same way you do with the server via Sonic Channel.

I'm interested in this, and thus I'm accepting forks then PRs to split things up in sonic-server and sonic-core sub-projects, both of them would live on crates.io (the current sonic implementation is already called sonic-server).

I'm thus eager to merge all of this into this repository it's done.

[Tails]

+1 for embedding Sonic. I have a use case for embedding it in a blockchain node server. If I would try to do that, I would duplicate the code that is now in main.rs and adjust it to run behind my Rocket.rs, but a more structural long-term approach would be better.

[rwv]

+1
I want to run sonic in Cloudflare workers Durable Objects. Since sonic is written in Rust and has low memory and cpu usage, it can be compiled to wasm and build a serverless search backend. Although I don't think this is a easy task.