[BEAM-11289] [Java] Integrate Google Cloud Recommendations AI functio… #13645

matthiasa4 · 2021-01-03T13:37:32Z

Integrate Google Cloud Recommendations AI functionality

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

Choose reviewer(s) and mention them in a comment (R: @username).
Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
Update CHANGES.md with noteworthy changes.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

Post-Commit Tests Status (on master branch)

Lang	Dataflow	Samza	Twister2
Go	---	---	---
Java
Python		---	---
XLang		---	---

Pre-Commit Tests Status (on master branch)

---	Java	Python	Go	Website	Whitespace	Typescript
Non-portable
Portable	---		---	---	---	---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

matthiasa4 · 2021-01-07T23:15:32Z

Quick question - due to the usage of GenericJsons in the transforms, I had to add 2 coders (DelegatingAtomicCoder.java & GenericJsonCoder.java). I don't think that the current folder makes sense to put these, but wasn't entirely sure if they belonged in sdks/java/core/src/main/java/org/apache/beam/sdk/coders either. What would you suggest?

pabloem · 2021-01-20T21:15:13Z

taking a look at this...

pabloem · 2021-01-20T23:58:28Z

to get passing RAT PreCommit, you need to add apache license headers to all new files : ) or excemptions for files that can't fit a comment with the license

matthiasa4 · 2021-02-06T13:32:28Z

@pabloem thanks - made some changes and looks like the tests are passing now :)

...ions/ml/src/main/java/org/apache/beam/sdk/extensions/ml/BatchRequestForRecommendationAI.java

...s/ml/src/main/java/org/apache/beam/sdk/extensions/ml/RecommendationAIImportCatalogItems.java

...ons/ml/src/main/java/org/apache/beam/sdk/extensions/ml/RecommendationAIImportUserEvents.java

…nality

pabloem

added a few comments

...ns/ml/src/main/java/org/apache/beam/sdk/extensions/ml/RecommendationAICreateCatalogItem.java

...tensions/ml/src/test/java/org/apache/beam/sdk/extensions/ml/RecommendationAIUserEventIT.java

pabloem · 2021-04-23T16:32:28Z

Run Java PostCommit

sdks/java/extensions/ml/build.gradle

...s/ml/src/main/java/org/apache/beam/sdk/extensions/ml/RecommendationAIImportCatalogItems.java

...ons/ml/src/main/java/org/apache/beam/sdk/extensions/ml/RecommendationAIImportUserEvents.java

matthiasa4 · 2021-04-28T12:00:59Z

hey @pabloem - I have addressed the comments we discussed, can you take another look please? :)

pabloem · 2021-04-28T18:51:57Z

Run Java PreCommit

matthiasa4 · 2021-04-28T20:13:01Z

@pabloem looks like those test failures are not related to this PR?

pabloem · 2021-04-29T23:53:23Z

Run Java PreCommit

pabloem

Sorry, I just added one more comment to remove public where possible, and to depend on BeamModulePlugin. All else looks good.

...s/ml/src/main/java/org/apache/beam/sdk/extensions/ml/RecommendationAIImportCatalogItems.java

...ons/ml/src/main/java/org/apache/beam/sdk/extensions/ml/RecommendationAIImportUserEvents.java

...sions/ml/src/main/java/org/apache/beam/sdk/extensions/ml/RecommendationAIWriteUserEvent.java

sdks/java/extensions/ml/build.gradle

github-actions · 2021-04-30T23:17:41Z

The Workflow run is cancelling this PR. It is an earlier duplicate of 2173354 run.

github-actions · 2021-04-30T23:17:44Z

The Workflow run is cancelling this PR. It is an earlier duplicate of 2173354 run.

pabloem · 2021-04-30T23:45:27Z

Run Java PostCommit

pabloem · 2021-04-30T23:47:02Z

Run PostCommit_Java_DataflowV2

pabloem · 2021-05-01T23:11:14Z

note that some tests are failing: https://ci-beam.apache.org/job/beam_PostCommit_Java_PR/680/

matthiasa4 · 2021-05-05T17:16:46Z

Hey @pabloem - the tests (for CatalogItem and UserEvent) work locally for me, but I am hitting my own GCP project (since the tests are calling APIs inside a project that needs the API enabled). I think this is why the tests are returning null values. Not sure if this is the only cause but is the apache-beam-testing project a real GCP project, does the service account that runs the test have access to it and are the APIs enabled on the other side?

For the Predict test I think the reason is related: since the model has no data yet (and we are trying to get a prediction based on recently_viewed_items) the resulting prediction returns empty. Either we need to put data in there or, alternatively, we could check if the faulty PCollection is empty. Wdyt?

Adjusting docs Adjusting prediction eventtype

matthiasa4 · 2021-05-12T20:40:07Z

Run Java PostCommit

matthiasa4 · 2021-05-13T13:28:15Z

@pabloem could it be that the serviceaccount running the tests doesn't have the correct permissions yet? (e.g. Recommendations AI Admin)

pabloem · 2021-05-19T17:21:04Z

Run Java PostCommit

aaltay · 2021-05-27T22:44:00Z

post commit test is still failing. - What is the next step on this PR?

matthiasa4 · 2021-05-28T16:52:57Z

@aaltay I would assume the same trouble as we have on the Python PR (service account permissions)

pabloem · 2021-06-07T22:37:13Z

Run Java PostCommit

matthiasa4 · 2021-06-13T15:24:12Z

Run Java PostCommit

matthiasa4 · 2021-06-13T16:21:16Z

@pabloem @aaltay looks like tests are passing now! Thanks for sorting permissions. Shall I squash & merge?

pabloem · 2021-06-18T21:01:35Z

sorry about the delay! - yes, I think you can squash and merge please @matthiasa4 : )

pabloem · 2021-06-18T21:01:54Z

I can do it if you prefer, but since you're a committer, feel free to merge it yourself. Thanks!

kennknowles · 2023-04-26T02:46:20Z

...sions/ml/src/main/java/org/apache/beam/sdk/extensions/ml/RecommendationAIWriteUserEvent.java

+  @Override
+  public PCollectionTuple expand(PCollection<GenericJson> input) {
+    return input.apply(
+        ParDo.of(new WriteUserEvent(projectId(), catalogName(), eventStore()))


Based on the usage here, it seems like projectId cannot actually be null. I see that it is explicitly nullable, though. Am I missing something?

Is it to allow the withXYZ pattern? And then we should validate the final state of everything at the beginning of expand?

probot-autolabeler bot added build extensions java labels Jan 3, 2021

matthiasa4 force-pushed the BEAM-11289-java-recommendation-ai branch from cbf20b2 to ff7fba0 Compare January 9, 2021 16:38

matthiasa4 force-pushed the BEAM-11289-java-recommendation-ai branch 4 times, most recently from 2a1fb72 to 51120a3 Compare January 25, 2021 08:29

aaltay marked this pull request as ready for review February 9, 2021 01:49

pabloem reviewed Feb 19, 2021

View reviewed changes

matthiasa4 added 2 commits February 28, 2021 21:39

[BEAM-11289] [Java] Integrate Google Cloud Recommendations AI functio…

a53b710

…nality

Introducing GroupIntoBatches-transform

c385c2c

matthiasa4 force-pushed the BEAM-11289-java-recommendation-ai branch from 51120a3 to c385c2c Compare February 28, 2021 22:38

matthiasa4 added 2 commits February 28, 2021 22:43

Apply Spotless

6928bf2

Update dependencies

092cb32

pabloem requested changes Mar 10, 2021

View reviewed changes

matthiasa4 added 3 commits April 18, 2021 22:57

Add toBuilder() methods

9c06a4b

Make projectId optional

4055175

Restructure classes and add RecommendationAIIO

cc193fd

pabloem reviewed Apr 23, 2021

View reviewed changes

matthiasa4 added 4 commits April 27, 2021 17:11

Update dependencies

0905e26

Add withShardedKey() to GroupIntoBatches

8c3eef0

Remove commented code

ebefbd9

Adding docstring to RecommendationAIIO

a8604af

matthiasa4 force-pushed the BEAM-11289-java-recommendation-ai branch from 880a80d to a8604af Compare April 27, 2021 19:00

matthiasa4 requested a review from pabloem April 28, 2021 12:00

pabloem reviewed Apr 29, 2021

View reviewed changes

matthiasa4 added 2 commits April 30, 2021 19:55

Updating modifiers

6670bc9

Updating dependencies

7ddf4fe

matthiasa4 requested a review from pabloem April 30, 2021 17:57

Fixing library inclusiuons

7e12f68

Fixing spotless

2b84c35

Adding productEventDetail for userEventIT

e9725bb

Adjusting docs Adjusting prediction eventtype

matthiasa4 merged commit c986168 into apache:master Jun 19, 2021

kennknowles reviewed Apr 26, 2023

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BEAM-11289] [Java] Integrate Google Cloud Recommendations AI functio… #13645

[BEAM-11289] [Java] Integrate Google Cloud Recommendations AI functio… #13645

matthiasa4 commented Jan 3, 2021 •

edited

matthiasa4 commented Jan 7, 2021

pabloem commented Jan 20, 2021

pabloem commented Jan 20, 2021

matthiasa4 commented Feb 6, 2021

pabloem left a comment

pabloem commented Apr 23, 2021

matthiasa4 commented Apr 28, 2021

pabloem commented Apr 28, 2021

matthiasa4 commented Apr 28, 2021

pabloem commented Apr 29, 2021

pabloem left a comment

github-actions bot commented Apr 30, 2021

github-actions bot commented Apr 30, 2021

pabloem commented Apr 30, 2021

pabloem commented Apr 30, 2021

pabloem commented May 1, 2021

matthiasa4 commented May 5, 2021

matthiasa4 commented May 12, 2021

matthiasa4 commented May 13, 2021

pabloem commented May 19, 2021

aaltay commented May 27, 2021

matthiasa4 commented May 28, 2021

pabloem commented Jun 7, 2021

matthiasa4 commented Jun 13, 2021

matthiasa4 commented Jun 13, 2021

pabloem commented Jun 18, 2021

pabloem commented Jun 18, 2021

kennknowles Apr 26, 2023

kennknowles Apr 26, 2023

[BEAM-11289] [Java] Integrate Google Cloud Recommendations AI functio… #13645

[BEAM-11289] [Java] Integrate Google Cloud Recommendations AI functio… #13645

Conversation

matthiasa4 commented Jan 3, 2021 • edited

Post-Commit Tests Status (on master branch)

Pre-Commit Tests Status (on master branch)

GitHub Actions Tests Status (on master branch)

matthiasa4 commented Jan 7, 2021

pabloem commented Jan 20, 2021

pabloem commented Jan 20, 2021

matthiasa4 commented Feb 6, 2021

pabloem left a comment

Choose a reason for hiding this comment

pabloem commented Apr 23, 2021

matthiasa4 commented Apr 28, 2021

pabloem commented Apr 28, 2021

matthiasa4 commented Apr 28, 2021

pabloem commented Apr 29, 2021

pabloem left a comment

Choose a reason for hiding this comment

github-actions bot commented Apr 30, 2021

github-actions bot commented Apr 30, 2021

pabloem commented Apr 30, 2021

pabloem commented Apr 30, 2021

pabloem commented May 1, 2021

matthiasa4 commented May 5, 2021

matthiasa4 commented May 12, 2021

matthiasa4 commented May 13, 2021

pabloem commented May 19, 2021

aaltay commented May 27, 2021

matthiasa4 commented May 28, 2021

pabloem commented Jun 7, 2021

matthiasa4 commented Jun 13, 2021

matthiasa4 commented Jun 13, 2021

pabloem commented Jun 18, 2021

pabloem commented Jun 18, 2021

kennknowles Apr 26, 2023

Choose a reason for hiding this comment

kennknowles Apr 26, 2023

Choose a reason for hiding this comment

matthiasa4 commented Jan 3, 2021 •

edited