[MINOR][CONNECT][DOC] Add information on how to regenerate proto for python client#41015
[MINOR][CONNECT][DOC] Add information on how to regenerate proto for python client#41015juliuszsompolski wants to merge 3 commits intoapache:masterfrom
Conversation
| @@ -21,6 +21,17 @@ user experience across all languages. Please follow the below guidelines: | |||
|
|
|||
| Python-specific development guidelines are located in [python/docs/source/development/testing.rst](https://github.com/apache/spark/blob/master/python/docs/source/development/testing.rst) that is published at [Development tab](https://spark.apache.org/docs/latest/api/python/development/index.html) in PySpark documentation. | |||
|
|
|||
There was a problem hiding this comment.
Would it be useful to add a README in this folder and link to this README?
[non blocking]
There was a problem hiding this comment.
I think that folder is rst based and published to the web documentation and not markdown based, so I think README.md doesn't fit there.
But I was surprised that the documentation there doesn't mention getting dependencies from dev/requirements.txt. But I'm not really familiar with python dev and these docs, maybe someone more familiar should have a better idea what would be the best place for it? @HyukjinKwon ?
Co-authored-by: Niranjan Jayakar <nija@databricks.com>
| Generate the Python files by running: | ||
|
|
||
| ``` | ||
| dev/connect-gen-protos.sh |
There was a problem hiding this comment.
Don't know is it specific to me but this command fails on my site with error:
+ buf generate --debug -vvv
...
Failure: 403 Forbidden
A workaround is to fork https://github.com/HyukjinKwon/my-github-actions and run the workflow:
and download the results:
|
+1, LGTM. Merging to master. |
MaxGekk
changed the title
| Install [buf](https://github.com/bufbuild/buf) | ||
|
|
||
| ``` | ||
| brew install bufbuild/buf/buf |
There was a problem hiding this comment.
| Generate the Python files by running: | ||
|
|
||
| ``` | ||
| dev/connect-gen-protos.sh |
There was a problem hiding this comment.
This is actually documented at https://spark.apache.org/docs/latest/api/python/development/testing.html#running-tests-for-python-client
There was a problem hiding this comment.
@juliuszsompolski was not able to find them. So I suspect they are not in a very discoverable spot.
Dedupe in a follow up sounds good. We can also change these to be links to the correct place.
| To generate the Python client code from the proto files: | ||
|
|
||
| First, make sure to have a Python environment with the installed dependencies. | ||
| Specifically, install `black` and dependencies from the "Spark Connect python proto generation plugin (optional)" section. |
There was a problem hiding this comment.
no biggie but I suspect we won't necessarily need to mention this
HyukjinKwon
left a comment
HyukjinKwon
left a comment
There was a problem hiding this comment.
LGTM. Let's clean up and dedup a little bit next time when we touch here.
…python client ### What changes were proposed in this pull request? Figuring out how to generate connect grpc proto in python was surprisingly hard to figure out for me (not knowing much about python development though), so adding it to the README. ### Why are the changes needed? Improving internal documentation. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Not applicable. Closes apache#41015 from juliuszsompolski/proto-regen-doc. Authored-by: Juliusz Sompolski <julek@databricks.com> Signed-off-by: Max Gekk <max.gekk@gmail.com>
5 participants
What changes were proposed in this pull request?
Figuring out how to generate connect grpc proto in python was surprisingly hard to figure out for me (not knowing much about python development though), so adding it to the README.
Why are the changes needed?
Improving internal documentation.
Does this PR introduce any user-facing change?
No.
How was this patch tested?
Not applicable.