Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix/protobuf missing field #22

Open
wants to merge 5 commits into
base: master
Choose a base branch
from
Open

Fix/protobuf missing field #22

wants to merge 5 commits into from

Conversation

tuteng
Copy link

@tuteng tuteng commented Mar 31, 2020

(If this PR fixes a github issue, please add Fixes #<xyz>.)

Fixes #

(or if this PR is one task of a github issue, please add Master Issue: #<xyz> to link to the master issue.)

Master Issue: #

Motivation

Explain here the context, and why you're making that change. What is the problem you're trying to solve.

Modifications

Describe the modifications you've done.

Verifying this change

  • Make sure that the change passes the CI checks.

(Please pick either of the following options)

This change is a trivial rework / code cleanup without any test coverage.

(or)

This change is already covered by existing tests, such as (please describe tests).

(or)

This change added tests and can be verified as follows:

(example:)

  • Added integration tests for end-to-end deployment with large payloads (10MB)
  • Extended integration test for recovery after broker failure

Does this pull request potentially affect one of the following parts:

If yes was chosen, please highlight the changes

  • Dependencies (does it add or upgrade a dependency): (yes / no)
  • The public API: (yes / no)
  • The schema: (yes / no / don't know)
  • The default values of configurations: (yes / no)
  • The wire protocol: (yes / no)
  • The rest endpoints: (yes / no)
  • The admin cli options: (yes / no)
  • Anything that affects deployment: (yes / no / don't know)

Documentation

  • Does this pull request introduce a new feature? (yes / no)
  • If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)
  • If a feature is not applicable for documentation, explain why?
  • If a feature is not documented yet in this PR, please create a followup issue for adding the documentation

@tuteng tuteng mentioned this pull request Mar 31, 2020
Merged
@tuteng tuteng force-pushed the fix/protobuf-missing-field branch from 6e1470c to 25ebb4a Compare March 31, 2020 10:21
tuteng added a commit to apache/pulsar that referenced this pull request Mar 31, 2020
@tuteng
[hotfix]Python function protobuf missing field and broker test failed (…
c955ff9 
…#6641)

### Motivation

At present, in the test, we found that due to the addition of a field `forwardSourceMessageProperty` in the proto file of function, this field was lost in the proto file generated by python and go. Due to python parsing with the following code:


```
json_format.Parse(args.function_details, function_details)
```

the following exception will be thrown. 

```
2020-03-30T13:13:25.2339031Z 13:13:24.379 [pulsar-external-listener-20-1] INFO  org.apache.pulsar.functions.runtime.process.ProcessRuntime - Started process successfully
2020-03-30T13:13:25.2339190Z Traceback (most recent call last):
2020-03-30T13:13:25.2340782Z   File "/pulsar/instances/python-instance/python_instance_main.py", line 211, in <module>
2020-03-30T13:13:25.2340944Z     main()
2020-03-30T13:13:25.2342589Z   File "/pulsar/instances/python-instance/python_instance_main.py", line 98, in main
2020-03-30T13:13:25.2342744Z     json_format.Parse(args.function_details, function_details)
2020-03-30T13:13:25.2354119Z   File "/usr/local/lib/python2.7/dist-packages/google/protobuf/json_format.py", line 430, in Parse
2020-03-30T13:13:25.2354284Z     return ParseDict(js, message, ignore_unknown_fields, descriptor_pool)
2020-03-30T13:13:25.2354689Z   File "/usr/local/lib/python2.7/dist-packages/google/protobuf/json_format.py", line 450, in ParseDict
2020-03-30T13:13:25.2354882Z     parser.ConvertMessage(js_dict, message)
2020-03-30T13:13:25.2355386Z   File "/usr/local/lib/python2.7/dist-packages/google/protobuf/json_format.py", line 481, in ConvertMessage
2020-03-30T13:13:25.2355537Z     self._ConvertFieldValuePair(value, message)
2020-03-30T13:13:25.2356082Z   File "/usr/local/lib/python2.7/dist-packages/google/protobuf/json_format.py", line 590, in _ConvertFieldValuePair
2020-03-30T13:13:25.2356559Z     raise ParseError('Failed to parse {0} field: {1}.'.format(name, e))
2020-03-30T13:13:25.2357199Z google.protobuf.json_format.ParseError: Failed to parse sink field: Message type "proto.SinkSpec" has no field named "forwardSourceMessageProperty".
2020-03-30T13:13:25.2357634Z  Available Fields(except extensions): ['className', 'configs', 'typeClassName', 'topic', 'serDeClassName', 'builtin', 'schemaType'].
```

This pr is mainly to fix the proto file generated by python first. In order to make the test pass smoothly, I will gradually fix this problem in other languages in the next pull request.


### Modifications

* Enable build docker image to pulsar and pulsar-all.
* Add new generated protobuf file for python
* Disable go function integration test
* Add sleep for method testGetPartitionedStatsInternal

### Verifying this change

The integration process test passed AmateurEvents#22
jiazhai pushed a commit to apache/pulsar that referenced this pull request May 8, 2020
@tuteng @jiazhai
[hotfix]Python function protobuf missing field and broker test failed (…
d39113a 
…#6641)

At present, in the test, we found that due to the addition of a field `forwardSourceMessageProperty` in the proto file of function, this field was lost in the proto file generated by python and go. Due to python parsing with the following code:

```
json_format.Parse(args.function_details, function_details)
```

the following exception will be thrown.

```
2020-03-30T13:13:25.2339031Z 13:13:24.379 [pulsar-external-listener-20-1] INFO  org.apache.pulsar.functions.runtime.process.ProcessRuntime - Started process successfully
2020-03-30T13:13:25.2339190Z Traceback (most recent call last):
2020-03-30T13:13:25.2340782Z   File "/pulsar/instances/python-instance/python_instance_main.py", line 211, in <module>
2020-03-30T13:13:25.2340944Z     main()
2020-03-30T13:13:25.2342589Z   File "/pulsar/instances/python-instance/python_instance_main.py", line 98, in main
2020-03-30T13:13:25.2342744Z     json_format.Parse(args.function_details, function_details)
2020-03-30T13:13:25.2354119Z   File "/usr/local/lib/python2.7/dist-packages/google/protobuf/json_format.py", line 430, in Parse
2020-03-30T13:13:25.2354284Z     return ParseDict(js, message, ignore_unknown_fields, descriptor_pool)
2020-03-30T13:13:25.2354689Z   File "/usr/local/lib/python2.7/dist-packages/google/protobuf/json_format.py", line 450, in ParseDict
2020-03-30T13:13:25.2354882Z     parser.ConvertMessage(js_dict, message)
2020-03-30T13:13:25.2355386Z   File "/usr/local/lib/python2.7/dist-packages/google/protobuf/json_format.py", line 481, in ConvertMessage
2020-03-30T13:13:25.2355537Z     self._ConvertFieldValuePair(value, message)
2020-03-30T13:13:25.2356082Z   File "/usr/local/lib/python2.7/dist-packages/google/protobuf/json_format.py", line 590, in _ConvertFieldValuePair
2020-03-30T13:13:25.2356559Z     raise ParseError('Failed to parse {0} field: {1}.'.format(name, e))
2020-03-30T13:13:25.2357199Z google.protobuf.json_format.ParseError: Failed to parse sink field: Message type "proto.SinkSpec" has no field named "forwardSourceMessageProperty".
2020-03-30T13:13:25.2357634Z  Available Fields(except extensions): ['className', 'configs', 'typeClassName', 'topic', 'serDeClassName', 'builtin', 'schemaType'].
```

This pr is mainly to fix the proto file generated by python first. In order to make the test pass smoothly, I will gradually fix this problem in other languages in the next pull request.

* Enable build docker image to pulsar and pulsar-all.
* Add new generated protobuf file for python
* Disable go function integration test
* Add sleep for method testGetPartitionedStatsInternal

The integration process test passed AmateurEvents#22
(cherry picked from commit c955ff9)
huangdx0726 pushed a commit to huangdx0726/pulsar that referenced this pull request Aug 24, 2020
@tuteng @huangdx0726
[hotfix]Python function protobuf missing field and broker test failed (…
bd8ac4b 
…apache#6641)

### Motivation

At present, in the test, we found that due to the addition of a field `forwardSourceMessageProperty` in the proto file of function, this field was lost in the proto file generated by python and go. Due to python parsing with the following code:


```
json_format.Parse(args.function_details, function_details)
```

the following exception will be thrown. 

```
2020-03-30T13:13:25.2339031Z 13:13:24.379 [pulsar-external-listener-20-1] INFO  org.apache.pulsar.functions.runtime.process.ProcessRuntime - Started process successfully
2020-03-30T13:13:25.2339190Z Traceback (most recent call last):
2020-03-30T13:13:25.2340782Z   File "/pulsar/instances/python-instance/python_instance_main.py", line 211, in <module>
2020-03-30T13:13:25.2340944Z     main()
2020-03-30T13:13:25.2342589Z   File "/pulsar/instances/python-instance/python_instance_main.py", line 98, in main
2020-03-30T13:13:25.2342744Z     json_format.Parse(args.function_details, function_details)
2020-03-30T13:13:25.2354119Z   File "/usr/local/lib/python2.7/dist-packages/google/protobuf/json_format.py", line 430, in Parse
2020-03-30T13:13:25.2354284Z     return ParseDict(js, message, ignore_unknown_fields, descriptor_pool)
2020-03-30T13:13:25.2354689Z   File "/usr/local/lib/python2.7/dist-packages/google/protobuf/json_format.py", line 450, in ParseDict
2020-03-30T13:13:25.2354882Z     parser.ConvertMessage(js_dict, message)
2020-03-30T13:13:25.2355386Z   File "/usr/local/lib/python2.7/dist-packages/google/protobuf/json_format.py", line 481, in ConvertMessage
2020-03-30T13:13:25.2355537Z     self._ConvertFieldValuePair(value, message)
2020-03-30T13:13:25.2356082Z   File "/usr/local/lib/python2.7/dist-packages/google/protobuf/json_format.py", line 590, in _ConvertFieldValuePair
2020-03-30T13:13:25.2356559Z     raise ParseError('Failed to parse {0} field: {1}.'.format(name, e))
2020-03-30T13:13:25.2357199Z google.protobuf.json_format.ParseError: Failed to parse sink field: Message type "proto.SinkSpec" has no field named "forwardSourceMessageProperty".
2020-03-30T13:13:25.2357634Z  Available Fields(except extensions): ['className', 'configs', 'typeClassName', 'topic', 'serDeClassName', 'builtin', 'schemaType'].
```

This pr is mainly to fix the proto file generated by python first. In order to make the test pass smoothly, I will gradually fix this problem in other languages in the next pull request.


### Modifications

* Enable build docker image to pulsar and pulsar-all.
* Add new generated protobuf file for python
* Disable go function integration test
* Add sleep for method testGetPartitionedStatsInternal

### Verifying this change

The integration process test passed AmateurEvents#22
sijie pushed a commit to apache/pulsar-release that referenced this pull request Nov 12, 2020
@tuteng
[hotfix]Python function protobuf missing field and broker test failed…
bc931c7 
… (#6641)

### Motivation

At present, in the test, we found that due to the addition of a field `forwardSourceMessageProperty` in the proto file of function, this field was lost in the proto file generated by python and go. Due to python parsing with the following code:


```
json_format.Parse(args.function_details, function_details)
```

the following exception will be thrown. 

```
2020-03-30T13:13:25.2339031Z 13:13:24.379 [pulsar-external-listener-20-1] INFO  org.apache.pulsar.functions.runtime.process.ProcessRuntime - Started process successfully
2020-03-30T13:13:25.2339190Z Traceback (most recent call last):
2020-03-30T13:13:25.2340782Z   File "/pulsar/instances/python-instance/python_instance_main.py", line 211, in <module>
2020-03-30T13:13:25.2340944Z     main()
2020-03-30T13:13:25.2342589Z   File "/pulsar/instances/python-instance/python_instance_main.py", line 98, in main
2020-03-30T13:13:25.2342744Z     json_format.Parse(args.function_details, function_details)
2020-03-30T13:13:25.2354119Z   File "/usr/local/lib/python2.7/dist-packages/google/protobuf/json_format.py", line 430, in Parse
2020-03-30T13:13:25.2354284Z     return ParseDict(js, message, ignore_unknown_fields, descriptor_pool)
2020-03-30T13:13:25.2354689Z   File "/usr/local/lib/python2.7/dist-packages/google/protobuf/json_format.py", line 450, in ParseDict
2020-03-30T13:13:25.2354882Z     parser.ConvertMessage(js_dict, message)
2020-03-30T13:13:25.2355386Z   File "/usr/local/lib/python2.7/dist-packages/google/protobuf/json_format.py", line 481, in ConvertMessage
2020-03-30T13:13:25.2355537Z     self._ConvertFieldValuePair(value, message)
2020-03-30T13:13:25.2356082Z   File "/usr/local/lib/python2.7/dist-packages/google/protobuf/json_format.py", line 590, in _ConvertFieldValuePair
2020-03-30T13:13:25.2356559Z     raise ParseError('Failed to parse {0} field: {1}.'.format(name, e))
2020-03-30T13:13:25.2357199Z google.protobuf.json_format.ParseError: Failed to parse sink field: Message type "proto.SinkSpec" has no field named "forwardSourceMessageProperty".
2020-03-30T13:13:25.2357634Z  Available Fields(except extensions): ['className', 'configs', 'typeClassName', 'topic', 'serDeClassName', 'builtin', 'schemaType'].
```

This pr is mainly to fix the proto file generated by python first. In order to make the test pass smoothly, I will gradually fix this problem in other languages in the next pull request.


### Modifications

* Enable build docker image to pulsar and pulsar-all.
* Add new generated protobuf file for python
* Disable go function integration test
* Add sleep for method testGetPartitionedStatsInternal

### Verifying this change

The integration process test passed AmateurEvents/pulsar#22
sijie pushed a commit to apache/pulsar-release that referenced this pull request Nov 12, 2020
@tuteng
[hotfix]Python function protobuf missing field and broker test failed…
d93d6e4 
… (#6641)

### Motivation

At present, in the test, we found that due to the addition of a field `forwardSourceMessageProperty` in the proto file of function, this field was lost in the proto file generated by python and go. Due to python parsing with the following code:


```
json_format.Parse(args.function_details, function_details)
```

the following exception will be thrown. 

```
2020-03-30T13:13:25.2339031Z 13:13:24.379 [pulsar-external-listener-20-1] INFO  org.apache.pulsar.functions.runtime.process.ProcessRuntime - Started process successfully
2020-03-30T13:13:25.2339190Z Traceback (most recent call last):
2020-03-30T13:13:25.2340782Z   File "/pulsar/instances/python-instance/python_instance_main.py", line 211, in <module>
2020-03-30T13:13:25.2340944Z     main()
2020-03-30T13:13:25.2342589Z   File "/pulsar/instances/python-instance/python_instance_main.py", line 98, in main
2020-03-30T13:13:25.2342744Z     json_format.Parse(args.function_details, function_details)
2020-03-30T13:13:25.2354119Z   File "/usr/local/lib/python2.7/dist-packages/google/protobuf/json_format.py", line 430, in Parse
2020-03-30T13:13:25.2354284Z     return ParseDict(js, message, ignore_unknown_fields, descriptor_pool)
2020-03-30T13:13:25.2354689Z   File "/usr/local/lib/python2.7/dist-packages/google/protobuf/json_format.py", line 450, in ParseDict
2020-03-30T13:13:25.2354882Z     parser.ConvertMessage(js_dict, message)
2020-03-30T13:13:25.2355386Z   File "/usr/local/lib/python2.7/dist-packages/google/protobuf/json_format.py", line 481, in ConvertMessage
2020-03-30T13:13:25.2355537Z     self._ConvertFieldValuePair(value, message)
2020-03-30T13:13:25.2356082Z   File "/usr/local/lib/python2.7/dist-packages/google/protobuf/json_format.py", line 590, in _ConvertFieldValuePair
2020-03-30T13:13:25.2356559Z     raise ParseError('Failed to parse {0} field: {1}.'.format(name, e))
2020-03-30T13:13:25.2357199Z google.protobuf.json_format.ParseError: Failed to parse sink field: Message type "proto.SinkSpec" has no field named "forwardSourceMessageProperty".
2020-03-30T13:13:25.2357634Z  Available Fields(except extensions): ['className', 'configs', 'typeClassName', 'topic', 'serDeClassName', 'builtin', 'schemaType'].
```

This pr is mainly to fix the proto file generated by python first. In order to make the test pass smoothly, I will gradually fix this problem in other languages in the next pull request.


### Modifications

* Enable build docker image to pulsar and pulsar-all.
* Add new generated protobuf file for python
* Disable go function integration test
* Add sleep for method testGetPartitionedStatsInternal

### Verifying this change

The integration process test passed AmateurEvents/pulsar#22
@github-actions
Copy link

The pr had no activity for 30 days, mark with Stale label.

@github-actions github-actions bot added the Stale label Sep 23, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
Stale
Projects
None yet
Milestone
No milestone
1 participant
@tuteng