Refresh Python module

[Dreamsorcerer]

The module seems so out of date, it didn't seem like breaking backwards compatibility would be a big deal.

Let me know if anything needs to change.

[Dreamsorcerer]

I'd note that this import doesn't appear to be used anymore, not sure what's happening there?

[Baliedge]

Thanks for the contribution! Have only constructive criticism to work out and then we can merge.

[Baliedge]

Can you move this stuff to a bash script in proto directory? And include a pip install for the tools.

And preferably we shouldn't need post-processing with sed commands. Looks like it's a matter of getting CWD or input paths right to fix that.

[Dreamsorcerer]

And preferably we shouldn't need post-processing with sed commands. Looks like it's a matter of getting CWD or input paths right to fix that.

Literally spent a couple of hours looking at this. In summary, it appears that Google have no interest in supporting modules which would require relative imports. It might be possible to get something to work by changing the proto directory, but I figured that might break things elsewhere. Feel free to have a go yourself though.

[Dreamsorcerer]

protocolbuffers/protobuf#1491

[Baliedge]

As you can probably tell, we don't use this Python functionality anymore. At least, not for Python 3. I'm thinking the entire Python client needs to be reimplemented from protos and disregard any consideration for legacy requirements.

I committed some changes to my branch for a genproto.sh script that already exists but needed some cleaning up. It does generate the code for Go and Python ok. There's some old unit tests in python/tests, but no idea how compatible that is to now current code. It fails, as you might expect.

[Dreamsorcerer]

The sed calls are not related to legacy (in fact, it apparently worked on Python 2 without rewriting the imports). It is because Google apparently just dumps all (thousands) of their files in a top-level directory that is on their PYTHONPATH.

For an installable package, we can't expect users to change their PYTHONPATH to point to the package's source code (I know that sound ridiculous, but I literally don't understand why the protobuf maintainers in that thread don't appear to understand that). So, sed rewrites the imports to import from the package (i.e. it could be done either as from gubernator or from .).

To test, run pip install . in your python directory, then (ensuring you're not in the source directory) open python and import gubernator. Without the sed changes you'll get an import error.

I'll go through the other changes next week.

[Dreamsorcerer]

Is this good to merge?

[thrawn01]

@Baliedge are we good with this PR? I'm about to break stuff, and it would be very nice to get this in before I do so.

[Baliedge]

@thrawn01 @Dreamsorcerer I was unable to verify the python changes in a docker container using provided unit test. Would like to see an automated test in Github Actions, but that could be a future PR.

Aside that, the merge conflicts need resolution.

docker run --rm -it -v $(pwd):/src -w /src ubuntu
apt update
apt install -y gcc g++ python3 python3-dev python3-pip
pip install pytest
pip install -r python/requirements.txt
pytest python

Relevant output:

root@ab3df8b79322:/src# pytest python
==================================== test session starts ====================================
platform linux -- Python 3.10.12, pytest-7.2.0, pluggy-1.2.0
rootdir: /src/python
collected 0 items / 1 error

========================================== ERRORS ===========================================
___________________________ ERROR collecting tests/test_client.py ___________________________
python/tests/test_client.py:22: in <module>
    from gubernator import (Algorithm, GetRateLimitsReq, HealthCheckReq, RateLimitReq,
python/gubernator/__init__.py:1: in <module>
    from gubernator.gubernator_pb2 import (
python/gubernator/gubernator_pb2.py:14: in <module>
    from google.api import annotations_pb2 as google_dot_api_dot_annotations__pb2
/usr/local/lib/python3.10/dist-packages/google/api/annotations_pb2.py:15: in <module>
    from google.api import http_pb2 as google_dot_api_dot_http__pb2
/usr/local/lib/python3.10/dist-packages/google/api/http_pb2.py:35: in <module>
    _descriptor.FieldDescriptor(
/usr/local/lib/python3.10/dist-packages/google/protobuf/descriptor.py:561: in __new__
    _message.Message._CheckCalledFromGeneratedFile()
E   TypeError: Descriptors cannot not be created directly.
E   If this call came from a _pb2.py file, your generated code is out of date and must be regenerated with protoc >= 3.19.0.
E   If you cannot immediately regenerate your protos, some other possible workarounds are:
E    1. Downgrade the protobuf package to 3.20.x or lower.
E    2. Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python (but this will use pure-Python parsing and will be much slower).
E
E   More information: https://developers.google.com/protocol-buffers/docs/news/2022-05-06#python-updates
================================== short test summary info ==================================
ERROR python/tests/test_client.py - TypeError: Descriptors cannot not be created directly.
!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!
===================================== 1 error in 0.14s ======================================

Please fix test as necessary.

[Dreamsorcerer]

Are you sure it's not using an old, previously installed, version of the code? That looks like one of the errors this PR fixes. Maybe try pip install . and rerun?

[Dreamsorcerer]

Also, the scripts have been deleted from the repository. How do you want to regenerate the files now? Shall I revert to the README?

[thrawn01]

So sorry about that @Dreamsorcerer, the scripts were removed because we are using https://buf.build/ now. I've been quite distracted in the past few months, and the next few weeks leading up to Black Friday will probably be even more distracted. Let me know what you need, and If you have some time to work on this PR, please ping me in the PR, and I'll make sure it gets merged.

[Dreamsorcerer]

@thrawn01 I can update the merge conflicts, but I don't know anything about buf. Does the Python build need to be added to buf somewhere? Or, should I just readd the build script for Python?

[Dreamsorcerer]

Or readd the README from before, so the steps can be run manually?

[thrawn01]

I can try to get buf working for Python, go ahead and re add the script, I take what is in the script and adapt it to buf.

[Dreamsorcerer]

Merged, should be ready again.

[Baliedge]

This requested task was completed as part of #209.
Additionally, the protobuf build script was replaced with the Buf tool.

[Dreamsorcerer]

This requested task was completed as part of #209. Additionally, the protobuf build script was replaced with the Buf tool.

Note this still won't work, as it doesn't have the change discussed above: #176 (comment)
If you try to publish a package to PyPI it will become immediately obvious.

[Baliedge]

Note this still won't work, as it doesn't have the change discussed above: #176 (comment) If you try to publish a package to PyPI it will become immediately obvious.

Sorry, I'm unable to give proper Python direction to support correct module structure. We don't use the Python client in our local usage of Gubernator, so I'm at the will of the public to provide direction there.

Because the protobuf build script process changed to "Buf", could you please start a new PR and apply your recommended changes from there?

[Dreamsorcerer]

Because the protobuf build script process changed to "Buf", could you please start a new PR and apply your recommended changes from there?

It's literally just the sed commands at the end, not sure how that integrates into your tooling now:
https://github.com/mailgun/gubernator/pull/176/files#diff-e58591b2cc09c4e870b65d2b260259412a84de0a76956598c86ee868a4174ebcR31-R34

Also, just noticed that the new one is missing the .pyi files for typing support. I added these with an extra couple of options to protoc:
https://github.com/mailgun/gubernator/pull/176/files#diff-e58591b2cc09c4e870b65d2b260259412a84de0a76956598c86ee868a4174ebcR27-R29