Skip to content

Commit

Permalink
Update example for Tokenizer and Filter API change
Browse files Browse the repository at this point in the history
  • Loading branch information
marcus-pousette committed Aug 12, 2022
1 parent 9d7a7fd commit 1b80161
Showing 1 changed file with 5 additions and 7 deletions.
12 changes: 5 additions & 7 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,10 +48,8 @@ use probly_search::{
};

// A white space tokenizer
fn tokenizer(s: &str) -> Vec<String> {
s.split(' ')
.map(|slice| slice.to_owned())
.collect::<Vec<String>>()
fn tokenizer(s: &str) -> Vec<&str> {
s.split(' ').collect::<Vec<_>>()
}

// We have to provide extraction functions for the fields we want to index
Expand All @@ -67,8 +65,8 @@ fn description_extract(d: &Doc) -> Option<&str> {
}

// A no-op filter
fn filter(s: &str) -> String {
s.to_owned()
fn filter(s: &str) -> &str {
s
}


Expand Down Expand Up @@ -163,4 +161,4 @@ assert_eq!(
Go through source tests in for the [BM25 implementation](https://github.com/quantleaf/probly-search/blob/master/src/query/score/default/bm25.rs) and [zero-to-one implementation](https://github.com/quantleaf/probly-search/blob/master/src/query/score/default/zero_to_one.rs) for more query examples.
## License

[MIT](http://opensource.org/licenses/MIT)
[MIT](http://opensource.org/licenses/MIT)

0 comments on commit 1b80161

Please sign in to comment.