OSM-POI: Include brand property [DRAFT]

[IritaSee]

In this PR, I wanted to solve issue #25 by creating a CSV file to list and also help the brand property matching process. This PR including:

• Compile a list of all brand names and operator names from the name-suggestion-index as a CSV with the columns id, display_name, and wiki_data in tmp folder.
  Deadline: 05.01.2022

• Create a PySpark UDF similar to the ones in the osm-poi/src/Processor.py that takes a string and returns the best match against the name-index-suggestions using fuzzy-wuzzy
  Deadline: 06.01.2022

• Apply that function for the brand and operator columns which creates two new columns brand_matched and operator_matched.
  Deadline: 07.01.2022

[mattigrthr]

The result is looking good! 👍🏽 Just added a few naming and coding convention comments.

[mattigrthr]

@IritaSee as discussed, here's some more background and assistance for task 3:

To add a new column to the PySpark data frame, use withColumn().
For example df_pois = df_pois.withColumn('brand_matched', match_brand_name(col('brand'))) would add a new column to df_pois called 'brand_matched' which would contain the value returned from the PySpark-UDF match_brand_name.

The UDF itself is applied for each row. So when you pass the col('brand') to the UDF it is working on all values individually (e.g., if there are 1,000 rows in that data frame, the UDF is executed 1,000 times). It effectively works on one string, e.g., 'McDonalds', and simply returns the best match from the brand name list.

Since the UDF is executed many times, we don't want to and can't read the CSV with the brand names a thousand times within the UDF. That's what 'broadcasts' in PySpark are for. They wrap a serializable value and make them available from inside the UDF. This is an example of how to use a broadcast from inside a UDF:

To make the main function more readable, you can create a static function that reads the CSV with the brand names, creates a broadcast from it, adds the two columns brand_matched and operator_matched by applying the UDF and returns the result.

The static function would be applied after the way, node, and relation data frames have been joined. It could look something like this: df_pois = Processor.match_brand_and_operator_names(df_pois)

[mattigrthr]

From looking and the code it seems basically ready. What is still missing @IritaSee?

[IritaSee]

From looking and the code it seems basically ready. What is still missing @IritaSee?

I feel not ready yet with the PR because of testing issue... I tested them all but I feel the branch did not run smoothly... I think we need to discuss about this but not for so long :") @mattigrthr

[mattigrthr]

This PR is currently on hold. See: #25 (comment)

After doing some initial tests with the brand and operator name matching, it turns out that including the matching in the OSM-POI pipeline directly would increase the runtime significantly. Therefore, we have decided to store the consolidated list of brand and operator names in a separate table in Postgres, which can then be used later in transformation blocks (e.g., on a filtered set of POIs and thus drastically reduce the runtime).

Since the canvas development currently has a higher priority for the core team, this issue is up for grabs again.