Skip to content

add language_field_trimming post processing script#135

Merged
missinglink merged 1 commit into
masterfrom
language_field_trimmin
Oct 14, 2020
Merged

add language_field_trimming post processing script#135
missinglink merged 1 commit into
masterfrom
language_field_trimmin

Conversation

@missinglink

@missinglink missinglink commented Oct 14, 2020

Copy link
Copy Markdown
Member

This PR adds a new 'post processing' script which aims to delete any names stored in language fields which are duplicated in the default language.

From the code comments:

/**
 * Language field post-processing script ensures that language tokens
 * present in the 'default' language are not duplicated in other languages.
 *
 * By default Pelias searches on the `name.default` field, and in some cases
 * it additionally searches on the language of the browser agent.
 *
 * This means that any name which exists in `name.default` need not additionally
 * exist in any of the other language fields.
 *
 * The benefits of this are that we can reduce the index size and any TF/IDF penalties.
 *
 * Example: the term 'Berlin' is indexed in *both* `name.default` and `name.de`.
 * In this case the `de` copy of the string 'Berlin' can be removed as it offers no value.
 */

This has the benefit of reducing the index size and also any TF/IDF penalties (or scoring boosts!) which may result in matching terms multiple times in different languages.

These names are not used for display (that's the job of the language service) so it should have no negative effect.

@Joxit you're most familiar with the language fields, does this look 👍 to you?

@missinglink missinglink requested a review from Joxit October 14, 2020 08:58
@missinglink

Copy link
Copy Markdown
Member Author

I'm going to try and reduce the amount of extra languages we're importing in another PR but this is a nice first step which should have no negatives.

Looking at something like https://raw.githubusercontent.com/whosonfirst-data/whosonfirst-data-admin-de/master/data/856/824/99/85682499.geojson the term "Berlin" appears many times in many languages, we don't need to index those other versions 🤷‍♂️

@missinglink missinglink force-pushed the language_field_trimmin branch from fd09745 to b83e155 Compare October 14, 2020 09:01
@missinglink

Copy link
Copy Markdown
Member Author

I found similar functionality in the WOF importer here: https://github.com/pelias/whosonfirst/blob/fee549816a8a29fc5c3daccc66129677f8d552d6/src/components/extractFields.js#L144

It doesn't hurt to have it twice but I think it's better to have it here since it applies to all sources including custom data (which was why the bug was originally reported)

@orangejulius orangejulius left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good!

The fact that we're already doing this in the WOF importer is a good sign, means this really won't affect much except venues in OSM with multiple names in different languages.

@Joxit Joxit left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In fact we are using name.{lang} in the API (especially for data from OSM) https://github.com/pelias/api/blob/b57bbd4ecdbc93c7546230a673dbddfc9fc5d110/middleware/changeLanguage.js#L70

But this is OK because AFAIK on OSM documents, name.default has a single value (from name=*) so the fallback is still OK 👍.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants