feat: add examples for asynchronous usage (callback, asycio)

[tomplus]

I added examples how to use a callback and how the library can be integrated with asyncio. Should I add this examples to readme too ?

Related to #323

[sebgoa]

Could you add an example with a Watch=True so the callbacks runs only when an event happens ?

thanks for working on this.

[tomplus]

I think callbacks don't work well with watch=True. API returns new lines on every change but a callback is called once when a server sends some response (at least headers). Am I right ?

I've prepared a modification to Watch() class to be able to add a callback which will be called on every change. I'll send PR to kubernetes-client/python-base with it soon.

[mbohlool]

@tomplus The new swagger codegen supports asyncio. We need to remove our manual changes to api_client.py file and then use the new version of generator. Then we can update this example and check it in. I will create an umbrella issue for that. Thanks for being patient.

[tomplus]

@mbohlool Thanks, I'll check the new swagger too.

[mbohlool]

@tomplus The master now does not have any manual changes in api_client.py. Can you checkout the asyncio python client in swagger-codegen? I assume it is a simple flag and you should be able to change our generator script to set the flag and try the client. As you are more familiar with this topic than me, you would be a good judge on this. Thanks.

[tomplus]

@mbohlool Sure, I'll do it over the weekend.

[tomplus]

I've regenerated library using swagger-codegen with switch --library asyncio.. I had to do some small changes. Likely there is some issue related to SSL in swagger - I'm going to confirm it and submit patch. Also urlib3 is replaced by aiohttp but module kube_config.py imports it.

Finally I have working piece of asynchronous code:

from kubernetes import client, config
import asyncio

async def main():
    # Configs can be set in Configuration class directly or using helper
    # utility. If no argument provided, the config will be loaded from
    # default location.
    config.load_kube_config()

    v1 = client.CoreV1Api()
    print("Listing pods with their IPs:")
    ret = await v1.list_pod_for_all_namespaces(watch=False)
    for i in ret.items:
        print("%s\t%s\t%s" % (i.status.pod_ip, i.metadata.namespace, i.metadata.name))


if __name__ == '__main__':
    loop = asyncio.get_event_loop()
    loop.run_until_complete(main())

I can submit my changes but I have one fundamental question. Using the asyncio library in swagger-codegen will generate new REST client based on aiohttp instead of urlib3. It means that we get completely new library... Do we want to start deploy new library ? I'm not sure. It would be better to have two different REST client inside one library.

Let me know what do you think .

[mbohlool]

We may want to start with having kubernetes-async library in addition to existing library to give this a try and fix problems with it and see what people think about it. We may eventually replace main library with it but I don't feel comfortable supporting async one yet so we should omit it from final releases for now.

@dims @sebgoa what do you think?

[dims]

@mbohlool agree with your assessment. We can experiment elsewhere and bring it back if it shows promise.

[achimnol]

I discovered the end of this thread too late. (did the same regeneration that @tomplus had already did)
So... How is the progress going here? 👀

[tomplus]

I'm working on it in my free time. There were some issues in swagger/asyncio generator which had to be fixed. I hope I'll publish async version in next month. Stay tuned...

[tomplus]

At last I’ve prepared the new library kubernetes_asyncio which is based on this Python client, but uses the asyncio generator from swagger-codegen. I've added a script to create this client to kubernetes-client/gen repo in this PR kubernetes-client/gen#60. I've also decided to incorporate functionality from sub-repository kubernetes-client/python-base because a lot of changes according to asyncio were needed.

Please take a look. Do you think it's possible to migrate this project to official kubernetes-clients' repo ?

[tomplus]

@mbohlool @sebgoa @dims @roycaihw Could you take a look? Thanks.

[holdenweb]

"Thank you" doesn't come often enough in the open source world. The small startup where I work bet on Kubernetes and Python as base server technologies eighteen months ago, and so far we haven't regretted it (and AWS have come on board with a managed service). Async operations bundled in will be a huge win for the Python community. So, thank you to everyone who's working on this development (or has done so).

[roycaihw]

@tomplus Thanks for working on this! The examples seem to be outdated (IIUC it's compatible with release-3.0), but we definitely want to give the asyncio library (https://github.com/tomplus/kubernetes_asyncio) a try and see what people think about it.

Asyncio is introduced in Python 3 and we need to support Python 2.7, so we have to keep the old synchronous library in this package. I'd suggest we having a kubernetes_asyncio library along with the existing kubernetes library, and letting users to switch import path to try the asyncio library. The two libraries shouldn't have any feature/user-visible difference other than the import path change.

The kubernetes_asyncio library depends on changes in python-base, but IMO maintaining two copies of python-base is not what we want. I have to challenge you to backport the change to python-base and make it generic for both libraries and minimize the difference.

We could have two new branches python:kubernetes_asyncio and python-base:kubernetes_asyncio (replacing the existing python-base:master submodule) checking out from existing masters for experimenting - seeing what changes are needed and releasing 7.0.0a2 version. New changes in master can be cherrypicked into these experimental branches.

@yliaog @mbohlool @dims @sebgoa what do you think?

[tomplus]

@roycaihw Many thanks for your response. kubernetes_asyncio is almost identical from user point of view (only changing imports and async syntax are required). I watch the official repository and catch up changes. It would be much easier if the library became experimental/official. Thanks for considering this issue.

Unfortunately there are a lot of changes related to classes from python-base. To be honest I had to rewrite Watch, Stream to be able to work in the asynchronous environment. We can try using the named branches but it may be confusing for contributors. I have other 2 options to discuss:

• Removing/merging python-base repository - I know it’s a controversial idea and I know the motivation, but I don’t know another library which uses python-base with other API groups. Is it still important for you? BTW only Python and Go use base-repos, other client libraries (Java, JS, C#, Haskell, Ruby) don’t have such base repos.

• I can create dedicated classes for asyncio with different names (like AsyncWatch, AsyncStream etc.) in python-base repo. This classes will be imported by kubernetes_asyncio and exported again with previous name. This trick allows us to have the common python-base and consistent libraries.

WRT final libraries, I think it’d be better to have libraries in separate repositories instead of branches, it’s more common to have a repository per package, it’s easy to install from repo, issues are tracked separately etc.

[txomon]

Hello guys, is there any way to push this forward?

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[tomplus]

/remove-lifecycle stale

[scottilee]

@tomplus is this still relevant? how should we move forward with this?

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[daggy1234]

/remove-lifecycle stale

[palnabarun]

/lifecycle frozen

[yliaog]

@tomplus I like your proposal for python-base below:

"I can create dedicated classes for asyncio with different names (like AsyncWatch, AsyncStream etc.) in python-base repo. This classes will be imported by kubernetes_asyncio and exported again with previous name. This trick allows us to have the common python-base and consistent libraries."

For async library, i'm thinking to have a separate directory inside python repo for asyncio, instead of a separate repo. then the releases / issues are all tracked in one repo. The current library and the async one should be kept independent, separate. of course we need to clearly state in the release note that the async library is new, and for preview.

The library has stopped supporting python 2 starting from v18.0.0. so the async library can be added in v18.0.0 or later.
https://github.com/kubernetes-client/python/blob/b13b350dba25e279765b3a8d055e5485b5289e2d/CHANGELOG.md#v18170a1

[roycaihw]

The library has stopped supporting python 2 starting from v18.0.0

it's pending #1413. The plan is to drop python 2 in v18.x.0b1. ETA is next Monday for v17 GA, and next week or two for v18.x.0b1.

[k8s-triage-robot]

The lifecycle/frozen label can not be applied to PRs.

This bot removes lifecycle/frozen from PRs because:

• Commenting /lifecycle frozen on a PR has not worked since March 2021
• PRs that remain open for >150 days are unlikely to be easily rebased

You can:

• Rebase this PR and attempt to get it merged
• Close this PR with /close

Please send feedback to sig-contributor-experience at kubernetes/community.

/remove-lifecycle frozen

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[remram44]

I am using kubernetes-asyncio successfully and I'm a bit confused about what is remaining in this issue (and why it's going "stale" while the requested features exists).

[yliaog]

@remram44

kubernetes-asyncio currently lives under tomplus's, the issue is to merge it to the github.com/kubernetes-client/python

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue or PR with /reopen
• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

[k8s-ci-robot]

@k8s-triage-robot: Closed this PR.

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue or PR with /reopen
• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[yliaog]

/reopen

[k8s-ci-robot]

@yliaog: Reopened this PR.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: tomplus
To complete the pull request process, please assign yliaog after the PR has been reviewed.
You can assign the PR to them by writing /assign @yliaog in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[yliaog]

@tomplus python-base repo has been merged into python. This should make it easier to add the asyncio to this repo.

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue or PR with /reopen
• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

[k8s-ci-robot]

@k8s-triage-robot: Closed this PR.

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue or PR with /reopen
• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[Ark-kun]

/reopen

[k8s-ci-robot]

@Ark-kun: You can't reopen an issue/PR unless you authored it or you are a collaborator.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[Skeen]

Any progress on this?