Skip to content

Commit

Permalink
[hotfix]Python function protobuf missing field and broker test failed (
Browse files Browse the repository at this point in the history
…#6641)

At present, in the test, we found that due to the addition of a field `forwardSourceMessageProperty` in the proto file of function, this field was lost in the proto file generated by python and go. Due to python parsing with the following code:

```
json_format.Parse(args.function_details, function_details)
```

the following exception will be thrown.

```
2020-03-30T13:13:25.2339031Z 13:13:24.379 [pulsar-external-listener-20-1] INFO  org.apache.pulsar.functions.runtime.process.ProcessRuntime - Started process successfully
2020-03-30T13:13:25.2339190Z Traceback (most recent call last):
2020-03-30T13:13:25.2340782Z   File "/pulsar/instances/python-instance/python_instance_main.py", line 211, in <module>
2020-03-30T13:13:25.2340944Z     main()
2020-03-30T13:13:25.2342589Z   File "/pulsar/instances/python-instance/python_instance_main.py", line 98, in main
2020-03-30T13:13:25.2342744Z     json_format.Parse(args.function_details, function_details)
2020-03-30T13:13:25.2354119Z   File "/usr/local/lib/python2.7/dist-packages/google/protobuf/json_format.py", line 430, in Parse
2020-03-30T13:13:25.2354284Z     return ParseDict(js, message, ignore_unknown_fields, descriptor_pool)
2020-03-30T13:13:25.2354689Z   File "/usr/local/lib/python2.7/dist-packages/google/protobuf/json_format.py", line 450, in ParseDict
2020-03-30T13:13:25.2354882Z     parser.ConvertMessage(js_dict, message)
2020-03-30T13:13:25.2355386Z   File "/usr/local/lib/python2.7/dist-packages/google/protobuf/json_format.py", line 481, in ConvertMessage
2020-03-30T13:13:25.2355537Z     self._ConvertFieldValuePair(value, message)
2020-03-30T13:13:25.2356082Z   File "/usr/local/lib/python2.7/dist-packages/google/protobuf/json_format.py", line 590, in _ConvertFieldValuePair
2020-03-30T13:13:25.2356559Z     raise ParseError('Failed to parse {0} field: {1}.'.format(name, e))
2020-03-30T13:13:25.2357199Z google.protobuf.json_format.ParseError: Failed to parse sink field: Message type "proto.SinkSpec" has no field named "forwardSourceMessageProperty".
2020-03-30T13:13:25.2357634Z  Available Fields(except extensions): ['className', 'configs', 'typeClassName', 'topic', 'serDeClassName', 'builtin', 'schemaType'].
```

This pr is mainly to fix the proto file generated by python first. In order to make the test pass smoothly, I will gradually fix this problem in other languages in the next pull request.

* Enable build docker image to pulsar and pulsar-all.
* Add new generated protobuf file for python
* Disable go function integration test
* Add sleep for method testGetPartitionedStatsInternal

The integration process test passed AmateurEvents#22
(cherry picked from commit c955ff9)
  • Loading branch information
tuteng authored and jiazhai committed May 8, 2020
1 parent c36170c commit d39113a
Show file tree
Hide file tree
Showing 4 changed files with 201 additions and 137 deletions.
8 changes: 8 additions & 0 deletions .github/workflows/ci-integration-process.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,14 @@ jobs:
if: steps.docs.outputs.changed_only == 'no'
run: mvn -B -f docker/pulsar-all/pom.xml install -am -Pdocker -DskipTests -Ddocker.nocache=true

- name: build artifacts and docker image
if: steps.docs.outputs.changed_only == 'no'
run: mvn -B -f docker/pulsar/pom.xml install -am -Pdocker -DskipTests -Ddocker.nocache=true

- name: build pulsar-all image
if: steps.docs.outputs.changed_only == 'no'
run: mvn -B -f docker/pulsar-all/pom.xml install -am -Pdocker -DskipTests -Ddocker.nocache=true

- name: build artifacts and docker image
run: mvn -B install -Dorg.slf4j.simpleLogger.defaultLogLevel=ERROR -Pdocker -DskipTests

Expand Down
8 changes: 8 additions & 0 deletions .github/workflows/ci-integration-thread.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,14 @@ jobs:
if: steps.docs.outputs.changed_only == 'no'
run: mvn -B -f docker/pulsar-all/pom.xml install -am -Pdocker -DskipTests -Ddocker.nocache=true

- name: build artifacts and docker image
if: steps.docs.outputs.changed_only == 'no'
run: mvn -B -f docker/pulsar/pom.xml install -am -Pdocker -DskipTests -Ddocker.nocache=true

- name: build pulsar-all image
if: steps.docs.outputs.changed_only == 'no'
run: mvn -B -f docker/pulsar-all/pom.xml install -am -Pdocker -DskipTests -Ddocker.nocache=true

- name: build artifacts and docker image
run: mvn -B install -Dorg.slf4j.simpleLogger.defaultLogLevel=ERROR -Pdocker -DskipTests

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -48,11 +48,13 @@
import java.util.Map;
import java.util.Optional;
import java.util.Set;
import java.util.UUID;
import java.util.concurrent.CompletableFuture;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.TimeUnit;

import javax.validation.constraints.AssertTrue;
import javax.ws.rs.client.InvocationCallback;
import javax.ws.rs.client.WebTarget;
import javax.ws.rs.core.Response.Status;
Expand Down
Loading

0 comments on commit d39113a

Please sign in to comment.