Refactor of Image Matching Algorithm #29
Conversation
Changes: * Adds spark udf * Modifies schema for top_candidates column to now view null image suggestions as an empty array * Saves output as parquet in hdfs
clarakosi
force-pushed
the
refactoring
branch
from
3053b67
to
9d5d259
Compare
Output is slightly modified now from an array of objects to that of an array of json. This change will need to be accounted for in the ETL pipeline
clarakosi
force-pushed
the
refactoring
branch
from
9d5d259
to
7259a13
Compare
clarakosi
changed the title
There was a problem hiding this comment.
LGTM!
I have a small comment re schema, that can be adressed once we move those bits of code out of this repo.
The notebook & udf refactoring look great! There's a couple of things that will need changes once we convert it to a script, but we can discuss those separarely.
| schema = ( | ||
| StructType() | ||
| .add("pandas_idx", StringType(), True) |
There was a problem hiding this comment.
Nice that we got rid of this!
FYI: these changes to spark & hql code will need to be incorporated into https://gitlab.wikimedia.org/gmodena/platform-airflow-dags/-/tree/multi-project-dags-repo/image-matching
There was a problem hiding this comment.
I'm actually not going to merge this PR into this repo but keep this one consistent with version one of the algorithm. I'll make PR on your GitLab repo
| @@ -62,11 +62,11 @@ def __init__(self, dataFrame: DataFrame): | |||
|
|
|||
| def transform(self) -> DataFrame: | |||
| with_recommendations = ( | |||
| self.dataFrame.where(~F.col("top_candidates").isNull()) | |||
| self.dataFrame.where(F.size(F.col("top_candidates")) > 0) | |||
There was a problem hiding this comment.
Here you removed a null check, but top_candidates schema says the column could be nullable.
I think your approach here is the correct one, but could we make the schema consistent?
Changes:
image suggestions as an empty array
Does not enable cluster mode in spark because it does not appear to be possible with jupyter notebooks