Add PatchResources to FhirIO. #14691

msbukal · 2021-04-29T21:43:28Z

References:
https://cloud.google.com/healthcare/docs/reference/rest/v1beta1/projects.locations.datasets.fhirStores.fhir/patch
https://cloud.google.com/healthcare/docs/how-tos/fhir-resources#patching_a_fhir_resource
https://cloud.google.com/healthcare/docs/how-tos/fhir-resources#conditionally_patching_a_fhir_resource

The Google Cloud FHIR service doesn't support neither Patch nor Conditional Patch within bundles, so it requires its own connector.

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

Choose reviewer(s) and mention them in a comment (R: @username).
Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
Update CHANGES.md with noteworthy changes.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

`ValidatesRunner` compliance status (on master branch)

Lang	ULR	Samza	Twister2
Go	---	---	---
Java
Python	---	---	---
XLang		---	---

Examples testing status on various runners

Lang	ULR	Dataflow	Flink	Samza	Spark	Twister2
Go	---	---	---	---	---	---	---
Java	---		---	---	---	---	---
Python	---	---	---	---	---	---	---
XLang	---	---	---	---	---	---	---

Post-Commit SDK/Transform Integration Tests Status (on master branch)

Go	Java	Python

Pre-Commit Tests Status (on master branch)

---	Java	Python	Go	Website	Whitespace	Typescript
Non-portable
Portable	---			---	---	---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

References: https://cloud.google.com/healthcare/docs/reference/rest/v1beta1/projects.locations.datasets.fhirStores.fhir/patch https://cloud.google.com/healthcare/docs/how-tos/fhir-resources#patching_a_fhir_resource https://cloud.google.com/healthcare/docs/how-tos/fhir-resources#conditionally_patching_a_fhir_resource The Google Cloud FHIR service doesn't support neither Patch nor Conditional Patch within bundles, so it requires it's own connector.

msbukal · 2021-04-30T14:22:35Z

R: @pabloem

pabloem · 2021-05-04T17:34:45Z

there's spotless errors: https://ci-beam.apache.org/job/beam_PreCommit_Spotless_Commit/15767/console

You can run the spotlessApply gradle task on the gradle package, and that should fix the spotless errors.

There's also a compile error: https://scans.gradle.com/s/y7kzzhs6dsx5g/console-log?task=:sdks:java:io:google-cloud-platform:compileJava

pabloem · 2021-05-04T18:30:03Z

also please use Beam's vendored guava : ) - https://scans.gradle.com/s/z7ynu5c4rnrb4/console-log?task=:sdks:java:io:google-cloud-platform:checkstyleTest

msbukal · 2021-05-04T18:39:06Z

I think everything should be fixed now :)

pabloem · 2021-05-04T21:52:26Z

Run Java PreCommit

pabloem

Added a couple comments.

...ava/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/healthcare/FhirIO.java

...e-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/healthcare/FhirPatchParameter.java

...-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/healthcare/HealthcareApiClient.java

pabloem

LGTM. Thanks! Just the one comment.

pabloem · 2021-05-05T23:16:06Z

...ava/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/healthcare/FhirIO.java

@@ -1417,8 +1432,107 @@ public void executeBundles(ProcessContext context) {
    }
  }

+  /** The type Patch resources. */
+  public static class PatchResources extends PTransform<PCollection<Input>, Write.Result> {
+


it might be good to add a non-public constructor here to ensure users won't create this transform directly, and they will use the FhirIO.patchResources directly. Thoughts?

I agree it makes sense, added a private constructor.

pabloem · 2021-05-06T17:23:54Z

Run Java PostCommit

pabloem · 2021-05-06T17:24:15Z

LGTM. I'll merge once postcommits are passing.

TheNeuralBit · 2021-05-07T16:31:37Z

It looks like somehow this broke beam_PostCommit_Java_DataflowV2 (BEAM-12310), could someone take a look?

pabloem · 2021-05-07T19:49:08Z

hm sorry about that. I missed running the Dataflow postcommits. @msbukal do you have time to take a look?

msbukal · 2021-05-07T20:09:31Z

In the failure linked, it looks like it's failing to read resources/ files on line:

https://github.com/msbukal/beam/blob/227b954fb86d2b31e298df6895984a41c49e3ef4/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/healthcare/FhirIOTestUtil.java#L67

I'm not sure how the failure could be introduced by this change, as I didn't modify this file at all. I also can't reproduce the error so not sure how to look further, is this a flake or a consistent failure?

pabloem · 2021-05-07T20:12:32Z

hmm here' sthe history of test suite runs: https://ci-beam.apache.org/job/beam_PostCommit_Java_DataflowV2/

TheNeuralBit · 2021-05-11T15:56:51Z

Can we try rolling back? It is a consistent failure.

Note that the failures are both in FhirIOPatchIT, which was added in this PR.

TheNeuralBit · 2021-05-11T15:59:44Z

For a repro, you will need to run on Dataflow v2, it seems fine on Dataflow v1. The simplest way is probably to just execute the same task used in the Jenkins job:

beam/.test-infra/jenkins/job_PostCommit_Java_DataflowV2.groovy

Line 42 in 79212a6

tasks(":runners:google-cloud-dataflow-java:postCommitRunnerV2")

msbukal · 2021-05-11T16:12:05Z

I get that it's coming from FhirIOPatchIT, but I can't find any differences between how FhirIOPatchIT and FhirIO[X]IT call
FhirIOTestUtil, so I'm not sure what's causing the initialization failure.

TheNeuralBit · 2021-05-11T16:32:24Z

Yeah I agree it's surprising, but since it's clear this PR broke a PostCommit we should roll it back first, then diagnose why. Please see https://beam.apache.org/contribute/postcommits-policies-details/#rollback_first

boyuanzz · 2021-05-13T22:04:25Z

From https://ci-beam.apache.org/job/beam_PostCommit_Java_DataflowV2/, we can see that the failure is consistent. Do you want to propose a rollback or you want me to help on that?

msbukal · 2021-05-14T15:26:58Z

Ah sorry, I wasn't aware that the rollback is something I am supposed to do. How do you make a rollback? I might not have the time to do this.

…irIO." This reverts commit e37bede

boyuanzz · 2021-05-14T19:27:20Z

Sent out revert PR: #14816

…#14691 from Add PatchResources to FhirIO." [BEAM-12310] Revert "Merge pull request #14691 from Add PatchResources to FhirIO."

msbukal added 2 commits May 4, 2021 13:50

Run spotless

186e15b

Fix @nullable tags

6e3764c

Fix guava import

201d24b

pabloem reviewed May 4, 2021

View reviewed changes

msbukal added 3 commits May 5, 2021 14:45

Prototype AutoValue implementation

4d16387

Final AutoValue implementation

cea4532

Add GroupIntoBatches to PatchResources.

0f720f3

pabloem reviewed May 5, 2021

View reviewed changes

Private constructor

227b954

pabloem merged commit e37bede into apache:master May 6, 2021

boyuanzz pushed a commit to boyuanzz/beam that referenced this pull request May 14, 2021

Revert "Merge pull request apache#14691 from Add PatchResources to Fh…

2ea61b7

…irIO." This reverts commit e37bede

boyuanzz mentioned this pull request May 14, 2021

[BEAM-12310] Revert "Merge pull request #14691 from Add PatchResources to FhirIO." #14816

Merged

4 tasks

boyuanzz added a commit that referenced this pull request May 15, 2021

Merge pull request #14816 from [BEAM-12310] Revert "Merge pull request …

d1cede0

…#14691 from Add PatchResources to FhirIO." [BEAM-12310] Revert "Merge pull request #14691 from Add PatchResources to FhirIO."

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add PatchResources to FhirIO. #14691

Add PatchResources to FhirIO. #14691

msbukal commented Apr 29, 2021

msbukal commented Apr 30, 2021

pabloem commented May 4, 2021

pabloem commented May 4, 2021

msbukal commented May 4, 2021

pabloem commented May 4, 2021

pabloem left a comment

pabloem left a comment

pabloem May 5, 2021

msbukal May 6, 2021

pabloem commented May 6, 2021

pabloem commented May 6, 2021

TheNeuralBit commented May 7, 2021

pabloem commented May 7, 2021

msbukal commented May 7, 2021

pabloem commented May 7, 2021

TheNeuralBit commented May 11, 2021

TheNeuralBit commented May 11, 2021

msbukal commented May 11, 2021

TheNeuralBit commented May 11, 2021

boyuanzz commented May 13, 2021

msbukal commented May 14, 2021 •

edited

Loading

boyuanzz commented May 14, 2021

Add PatchResources to FhirIO. #14691

Add PatchResources to FhirIO. #14691

Conversation

msbukal commented Apr 29, 2021

ValidatesRunner compliance status (on master branch)

Examples testing status on various runners

Post-Commit SDK/Transform Integration Tests Status (on master branch)

Pre-Commit Tests Status (on master branch)

GitHub Actions Tests Status (on master branch)

msbukal commented Apr 30, 2021

pabloem commented May 4, 2021

pabloem commented May 4, 2021

msbukal commented May 4, 2021

pabloem commented May 4, 2021

pabloem left a comment

Choose a reason for hiding this comment

pabloem left a comment

Choose a reason for hiding this comment

pabloem May 5, 2021

Choose a reason for hiding this comment

msbukal May 6, 2021

Choose a reason for hiding this comment

pabloem commented May 6, 2021

pabloem commented May 6, 2021

TheNeuralBit commented May 7, 2021

pabloem commented May 7, 2021

msbukal commented May 7, 2021

pabloem commented May 7, 2021

TheNeuralBit commented May 11, 2021

TheNeuralBit commented May 11, 2021

msbukal commented May 11, 2021

TheNeuralBit commented May 11, 2021

boyuanzz commented May 13, 2021

msbukal commented May 14, 2021 • edited Loading

boyuanzz commented May 14, 2021

`ValidatesRunner` compliance status (on master branch)

msbukal commented May 14, 2021 •

edited

Loading