Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Re-enable bundling (for consideration) #1950

Merged
merged 2 commits into from Jul 1, 2016
Merged

Re-enable bundling (for consideration) #1950

merged 2 commits into from Jul 1, 2016

Conversation

daspecster
Copy link
Contributor

I know this is probably up for debate, I just wanted to learn more about this.
If you guys do want to re-enable this(#1911) I'll make whatever changes are needed!

Or let me know and I'll close this if you want to go in a different direction.

@googlebot googlebot added the cla: yes This human has signed the Contributor License Agreement. label Jul 1, 2016
@daspecster daspecster added the api: pubsub Issues related to the Pub/Sub API. label Jul 1, 2016
@tseaver tseaver added the grpc label Jul 1, 2016
tseaver
tseaver reviewed Jul 1, 2016
View reviewed changes
gcloud/pubsub/_gax.py
result = self._gax_api.publish(topic_path, message_pbs,
options=options)
# result = self._gax_api.publish(topic_path, message_pbs,
# options=options)

This comment was marked as spam.

Sign in to view

This comment was marked as spam.

Sign in to view

@tseaver
Copy link
Contributor

tseaver commented Jul 1, 2016

@daspecster Can you try running the pubsub system tests on your branch with gRPC enabled? E.g.:

$ tox -e system-tests --notest
$ GCLOUD_ENABLE_GAX .tox/system-tests/bin/python \
  system_tests/run_system_test.py --package=pubsub

@tseaver
Copy link
Contributor

tseaver commented Jul 1, 2016

LGTM once Travis passes, assuming the system tests pass for you with gRPC/GAX enabled.

@daspecster
Copy link
Contributor Author

@tseaver, not sure what all this leaked stuff means but the system tests seemed to have passed.

D0701 12:21:24.977269000 140735197470720 ev_posix.c:101] Using polling engine: poll
test_create_subscription_defaults (pubsub.TestPubsub) ... ok
test_create_subscription_w_ack_deadline (pubsub.TestPubsub) ... ok
test_create_topic (pubsub.TestPubsub) ... ok
test_fetch_delete_subscription_w_deleted_topic (pubsub.TestPubsub) ... ok
test_list_subscriptions (pubsub.TestPubsub) ... ok
test_list_topics (pubsub.TestPubsub) ... ok
test_message_pull_mode_e2e (pubsub.TestPubsub) ... ok
test_subscription_iam_policy (pubsub.TestPubsub) ... ok
test_topic_iam_policy (pubsub.TestPubsub) ... ok

----------------------------------------------------------------------
Ran 9 tests in 92.143s

OK
D0701 12:22:58.576802000 140735197470720 iomgr.c:99] Waiting for 2 iomgr objects to be destroyed
D0701 12:22:59.605587000 140735197470720 iomgr.c:99] Waiting for 2 iomgr objects to be destroyed
D0701 12:23:00.633029000 140735197470720 iomgr.c:99] Waiting for 2 iomgr objects to be destroyed
D0701 12:23:01.661054000 140735197470720 iomgr.c:99] Waiting for 2 iomgr objects to be destroyed
D0701 12:23:02.677249000 140735197470720 iomgr.c:99] Waiting for 2 iomgr objects to be destroyed
D0701 12:23:03.704005000 140735197470720 iomgr.c:99] Waiting for 2 iomgr objects to be destroyed
D0701 12:23:04.734016000 140735197470720 iomgr.c:99] Waiting for 2 iomgr objects to be destroyed
D0701 12:23:05.769014000 140735197470720 iomgr.c:99] Waiting for 2 iomgr objects to be destroyed
D0701 12:23:06.795325000 140735197470720 iomgr.c:99] Waiting for 2 iomgr objects to be destroyed
D0701 12:23:07.616885000 140735197470720 iomgr.c:118] Failed to free 2 iomgr objects before shutdown deadline: memory leaks are likely
D0701 12:23:07.616913000 140735197470720 iomgr.c:80] LEAKED OBJECT: tcp-client:ipv4:216.58.217.10:443 fd=10 0x7f8eea7435c0
D0701 12:23:07.616921000 140735197470720 iomgr.c:80] LEAKED OBJECT: tcp-client:ipv4:216.58.217.10:443 fd=14 0x7f8eecb966a0
D0701 12:23:07.616955000 140735197470720 metadata.c:238] WARNING: 2 metadata elements were leaked
D0701 12:23:07.616964000 140735197470720 metadata.c:238] WARNING: 7 metadata elements were leaked
D0701 12:23:07.617015000 140735197470720 metadata.c:238] WARNING: 2 metadata elements were leaked
D0701 12:23:07.617024000 140735197470720 metadata.c:238] WARNING: 3 metadata elements were leaked
D0701 12:23:07.617030000 140735197470720 metadata.c:238] WARNING: 4 metadata elements were leaked
D0701 12:23:07.617035000 140735197470720 metadata.c:238] WARNING: 2 metadata elements were leaked
D0701 12:23:07.617041000 140735197470720 metadata.c:238] WARNING: 3 metadata elements were leaked
D0701 12:23:07.617044000 140735197470720 metadata.c:238] WARNING: 9 metadata elements were leaked
D0701 12:23:07.617047000 140735197470720 metadata.c:238] WARNING: 6 metadata elements were leaked
D0701 12:23:07.617052000 140735197470720 metadata.c:238] WARNING: 4 metadata elements were leaked
D0701 12:23:07.617054000 140735197470720 metadata.c:238] WARNING: 3 metadata elements were leaked
D0701 12:23:07.617058000 140735197470720 metadata.c:238] WARNING: 7 metadata elements were leaked
D0701 12:23:07.617061000 140735197470720 metadata.c:238] WARNING: 8 metadata elements were leaked
D0701 12:23:07.617065000 140735197470720 metadata.c:238] WARNING: 4 metadata elements were leaked
D0701 12:23:07.617122000 140735197470720 metadata.c:238] WARNING: 4 metadata elements were leaked
D0701 12:23:07.617130000 140735197470720 metadata.c:238] WARNING: 4 metadata elements were leaked
D0701 12:23:07.617136000 140735197470720 metadata.c:251] WARNING: 2 metadata strings were leaked
D0701 12:23:07.617141000 140735197470720 metadata.c:255] LEAKED: /google.pubsub.v1.Publisher/DeleteTopic
D0701 12:23:07.617146000 140735197470720 metadata.c:255] LEAKED: /google.pubsub.v1.Subscriber/CreateSubscription
D0701 12:23:07.617152000 140735197470720 metadata.c:251] WARNING: 3 metadata strings were leaked
D0701 12:23:07.617158000 140735197470720 metadata.c:255] LEAKED: gax/0.12.2 gapic/0.1.0 gax/0.12.2 python/2.7.10
D0701 12:23:07.617163000 140735197470720 metadata.c:255] LEAKED: Fri, 01 Jul 2016 18:21:59 GMT
D0701 12:23:07.617167000 140735197470720 metadata.c:255] LEAKED: alt-svc
D0701 12:23:07.617173000 140735197470720 metadata.c:251] WARNING: 1 metadata strings were leaked
D0701 12:23:07.617177000 140735197470720 metadata.c:255] LEAKED: Fri, 01 Jul 2016 18:22:09 GMT
D0701 12:23:07.617182000 140735197470720 metadata.c:251] WARNING: 2 metadata strings were leaked
D0701 12:23:07.617187000 140735197470720 metadata.c:255] LEAKED: 5
D0701 12:23:07.617192000 140735197470720 metadata.c:255] LEAKED: x-goog-api-client
D0701 12:23:07.617197000 140735197470720 metadata.c:251] WARNING: 4 metadata strings were leaked
D0701 12:23:07.617202000 140735197470720 metadata.c:255] LEAKED: Fri, 01 Jul 2016 18:22:57 GMT
D0701 12:23:07.617207000 140735197470720 metadata.c:255] LEAKED: Fri, 01 Jul 2016 18:22:43 GMT
D0701 12:23:07.617211000 140735197470720 metadata.c:255] LEAKED: Fri, 01 Jul 2016 18:22:15 GMT
D0701 12:23:07.617216000 140735197470720 metadata.c:255] LEAKED: Fri, 01 Jul 2016 18:21:33 GMT
D0701 12:23:07.617220000 140735197470720 metadata.c:251] WARNING: 1 metadata strings were leaked
D0701 12:23:07.617225000 140735197470720 metadata.c:255] LEAKED: Fri, 01 Jul 2016 18:21:50 GMT
D0701 12:23:07.617229000 140735197470720 metadata.c:251] WARNING: 3 metadata strings were leaked
D0701 12:23:07.617234000 140735197470720 metadata.c:255] LEAKED: 443:quic
D0701 12:23:07.617238000 140735197470720 metadata.c:255] LEAKED: Fri, 01 Jul 2016 18:21:42 GMT
D0701 12:23:07.617243000 140735197470720 metadata.c:255] LEAKED: 29500m
D0701 12:23:07.617248000 140735197470720 metadata.c:251] WARNING: 2 metadata strings were leaked
D0701 12:23:07.617252000 140735197470720 metadata.c:255] LEAKED: Fri, 01 Jul 2016 18:21:29 GMT
D0701 12:23:07.617258000 140735197470720 metadata.c:255] LEAKED: Resource not found (resource=subscribe-me-1467397285328).
D0701 12:23:07.617659000 140735197470720 metadata.c:251] WARNING: 1 metadata strings were leaked
D0701 12:23:07.617669000 140735197470720 metadata.c:255] LEAKED: /google.pubsub.v1.Publisher/ListTopicSubscriptions
D0701 12:23:07.617675000 140735197470720 metadata.c:251] WARNING: 2 metadata strings were leaked
D0701 12:23:07.617681000 140735197470720 metadata.c:255] LEAKED: Fri, 01 Jul 2016 18:22:46 GMT
D0701 12:23:07.617685000 140735197470720 metadata.c:255] LEAKED: Fri, 01 Jul 2016 18:22:10 GMT
D0701 12:23:07.617691000 140735197470720 metadata.c:251] WARNING: 2 metadata strings were leaked
D0701 12:23:07.617698000 140735197470720 metadata.c:255] LEAKED: Fri, 01 Jul 2016 18:22:53 GMT
D0701 12:23:07.617703000 140735197470720 metadata.c:255] LEAKED: /google.pubsub.v1.Subscriber/DeleteSubscription
D0701 12:23:07.617710000 140735197470720 metadata.c:251] WARNING: 1 metadata strings were leaked
D0701 12:23:07.617715000 140735197470720 metadata.c:255] LEAKED: quic=":443"; ma=2592000; v="34,33,32,31,30,29,28,27,26,25"
D0701 12:23:07.617722000 140735197470720 metadata.c:251] WARNING: 3 metadata strings were leaked
D0701 12:23:07.617727000 140735197470720 metadata.c:255] LEAKED: Fri, 01 Jul 2016 18:22:14 GMT
D0701 12:23:07.617731000 140735197470720 metadata.c:255] LEAKED: Python-gRPC-0.15.0 grpc-c/0.15.0 (osx; chttp2)
D0701 12:23:07.617736000 140735197470720 metadata.c:255] LEAKED: Fri, 01 Jul 2016 18:21:49 GMT
D0701 12:23:07.617742000 140735197470720 metadata.c:251] WARNING: 2 metadata strings were leaked
D0701 12:23:07.617747000 140735197470720 metadata.c:255] LEAKED: Fri, 01 Jul 2016 18:21:28 GMT
D0701 12:23:07.617751000 140735197470720 metadata.c:255] LEAKED: Fri, 01 Jul 2016 18:22:50 GMT
D0701 12:23:07.617811000 140735197470720 metadata.c:251] WARNING: 3 metadata strings were leaked
D0701 12:23:07.617817000 140735197470720 metadata.c:255] LEAKED: 30S
D0701 12:23:07.617819000 140735197470720 metadata.c:255] LEAKED: Fri, 01 Jul 2016 18:21:26 GMT
D0701 12:23:07.617823000 140735197470720 metadata.c:255] LEAKED: Resource not found (resource=subscribing-now_1467397296399).
D0701 12:23:07.617839000 140735197470720 metadata.c:251] WARNING: 3 metadata strings were leaked
D0701 12:23:07.617844000 140735197470720 metadata.c:255] LEAKED: Fri, 01 Jul 2016 18:22:45 GMT
D0701 12:23:07.617851000 140735197470720 metadata.c:255] LEAKED: Resource not found (resource=subscribing-now-1467397338506).
D0701 12:23:07.617857000 140735197470720 metadata.c:255] LEAKED: Fri, 01 Jul 2016 18:22:04 GMT
D0701 12:23:07.617863000 140735197470720 metadata.c:251] WARNING: 3 metadata strings were leaked
D0701 12:23:07.617868000 140735197470720 metadata.c:255] LEAKED: Fri, 01 Jul 2016 18:22:02 GMT
D0701 12:23:07.617873000 140735197470720 metadata.c:255] LEAKED: attachment
D0701 12:23:07.617877000 140735197470720 metadata.c:255] LEAKED: Fri, 01 Jul 2016 18:22:56 GMT
D0701 12:23:07.617883000 140735197470720 metadata.c:251] WARNING: 2 metadata strings were leaked
D0701 12:23:07.617889000 140735197470720 metadata.c:255] LEAKED: Fri, 01 Jul 2016 18:21:47 GMT
D0701 12:23:07.617894000 140735197470720 metadata.c:255] LEAKED: Fri, 01 Jul 2016 18:22:07 GMT
D0701 12:23:07.617900000 140735197470720 metadata.c:251] WARNING: 2 metadata strings were leaked
D0701 12:23:07.617905000 140735197470720 metadata.c:255] LEAKED: Fri, 01 Jul 2016 18:21:31 GMT
D0701 12:23:07.617911000 140735197470720 metadata.c:255] LEAKED: /google.pubsub.v1.Publisher/CreateTopic
D0701 12:23:07.617917000 140735197470720 metadata.c:251] WARNING: 1 metadata strings were leaked
D0701 12:23:07.617921000 140735197470720 metadata.c:255] LEAKED: Fri, 01 Jul 2016 18:21:38 GMT
D0701 12:23:07.617926000 140735197470720 metadata.c:251] WARNING: 3 metadata strings were leaked
D0701 12:23:07.617931000 140735197470720 metadata.c:255] LEAKED: Fri, 01 Jul 2016 18:21:36 GMT
D0701 12:23:07.617935000 140735197470720 metadata.c:255] LEAKED: Fri, 01 Jul 2016 18:21:34 GMT
D0701 12:23:07.617941000 140735197470720 metadata.c:255] LEAKED: Bearer ya29.Ci8SA6L-5x86Grpc9fbdTCm9cHO5iFZYal9E36MePKdkOCcHqKAV5zLROpRRColPsQ
D0701 12:23:07.617948000 140735197470720 metadata.c:251] WARNING: 2 metadata strings were leaked
D0701 12:23:07.617953000 140735197470720 metadata.c:255] LEAKED: /google.pubsub.v1.Publisher/ListTopics
D0701 12:23:07.617973000 140735197470720 metadata.c:255] LEAKED: Fri, 01 Jul 2016 18:22:48 GMT
D0701 12:23:07.617979000 140735197470720 metadata.c:251] WARNING: 2 metadata strings were leaked
D0701 12:23:07.617984000 140735197470720 metadata.c:255] LEAKED: Fri, 01 Jul 2016 18:22:12 GMT
D0701 12:23:07.617988000 140735197470720 metadata.c:255] LEAKED: alternate-protocol
D0701 12:23:07.617993000 140735197470720 metadata.c:251] WARNING: 3 metadata strings were leaked
D0701 12:23:07.617998000 140735197470720 metadata.c:255] LEAKED: /google.pubsub.v1.Subscriber/GetSubscription
D0701 12:23:07.618003000 140735197470720 metadata.c:255] LEAKED: pubsub.googleapis.com:443
D0701 12:23:07.618007000 140735197470720 metadata.c:255] LEAKED: Fri, 01 Jul 2016 18:21:39 GMT
D0701 12:23:07.618012000 140735197470720 metadata.c:251] WARNING: 6 metadata strings were leaked
D0701 12:23:07.618017000 140735197470720 metadata.c:255] LEAKED: /google.pubsub.v1.Publisher/Publish
D0701 12:23:07.618022000 140735197470720 metadata.c:255] LEAKED: Fri, 01 Jul 2016 18:22:16 GMT
D0701 12:23:07.618061000 140735197470720 metadata.c:255] LEAKED: Fri, 01 Jul 2016 18:21:44 GMT
D0701 12:23:07.618067000 140735197470720 metadata.c:255] LEAKED: Fri, 01 Jul 2016 18:22:31 GMT
D0701 12:23:07.618071000 140735197470720 metadata.c:255] LEAKED: Fri, 01 Jul 2016 18:21:51 GMT
D0701 12:23:07.618076000 140735197470720 metadata.c:255] LEAKED: Fri, 01 Jul 2016 18:22:05 GMT
D0701 12:23:07.618082000 140735197470720 metadata.c:251] WARNING: 4 metadata strings were leaked
D0701 12:23:07.618086000 140735197470720 metadata.c:255] LEAKED: /google.pubsub.v1.Subscriber/Acknowledge
D0701 12:23:07.618092000 140735197470720 metadata.c:255] LEAKED: Bearer ya29.Ci8SA7K6GmtEnF0QO5LtiTYPM7yhhh4SSDI91OOEh_GyWucwz_KXxT1PX4c7cUzJZw
D0701 12:23:07.618097000 140735197470720 metadata.c:255] LEAKED: Fri, 01 Jul 2016 18:22:42 GMT
D0701 12:23:07.618102000 140735197470720 metadata.c:255] LEAKED: Fri, 01 Jul 2016 18:21:54 GMT
D0701 12:23:07.618107000 140735197470720 metadata.c:251] WARNING: 2 metadata strings were leaked
D0701 12:23:07.618112000 140735197470720 metadata.c:255] LEAKED: /google.pubsub.v1.Publisher/GetTopic
D0701 12:23:07.618117000 140735197470720 metadata.c:255] LEAKED: Fri, 01 Jul 2016 18:22:18 GMT
D0701 12:23:07.618122000 140735197470720 metadata.c:251] WARNING: 2 metadata strings were leaked
D0701 12:23:07.618127000 140735197470720 metadata.c:255] LEAKED: /google.pubsub.v1.Subscriber/Pull
D0701 12:23:07.618131000 140735197470720 metadata.c:255] LEAKED: Fri, 01 Jul 2016 18:22:55 GMT
D0701 12:23:07.618136000 140735197470720 metadata.c:251] WARNING: 1 metadata strings were leaked
D0701 12:23:07.618141000 140735197470720 metadata.c:255] LEAKED: Resource not found (resource=a-new-topic-1467397300700).
D0701 12:23:07.618146000 140735197470720 metadata.c:251] WARNING: 5 metadata strings were leaked
D0701 12:23:07.618151000 140735197470720 metadata.c:255] LEAKED: /google.pubsub.v1.Subscriber/ListSubscriptions
D0701 12:23:07.618156000 140735197470720 metadata.c:255] LEAKED: Fri, 01 Jul 2016 18:21:40 GMT
D0701 12:23:07.618161000 140735197470720 metadata.c:255] LEAKED: Resource not found (resource=subscribing-now-1467397288719).
D0701 12:23:07.618166000 140735197470720 metadata.c:255] LEAKED: Fri, 01 Jul 2016 18:22:21 GMT
D0701 12:23:07.618170000 140735197470720 metadata.c:255] LEAKED: Fri, 01 Jul 2016 18:21:53 GMT
D0701 12:23:07.618176000 140735197470720 metadata.c:251] WARNING: 2 metadata strings were leaked
D0701 12:23:07.618180000 140735197470720 metadata.c:255] LEAKED: Fri, 01 Jul 2016 18:22:06 GMT
D0701 12:23:07.618185000 140735197470720 metadata.c:255] LEAKED: Fri, 01 Jul 2016 18:21:56 GM

@daspecster
Copy link
Contributor Author

I ran it with this...

GCLOUD_TESTS_PROJECT_ID=ferrous-arena GOOGLE_APPLICATION_CREDENTIALS=creds.json GCLOUD_ENABLE_GAX=True .tox/system-tests/bin/python system_tests/run_system_test.py --package=pubsub

@tseaver
Copy link
Contributor

tseaver commented Jul 1, 2016

That spew at the end is gRPC whining: harmless, but annoying for our purposes. I won't tackle #1873 until it is resolved, though.

@tseaver
Copy link
Contributor

tseaver commented Jul 1, 2016

LGTM

@daspecster daspecster merged commit 628ee9a into googleapis:master Jul 1, 2016
@daspecster daspecster deleted the enable-pubsub-bundling branch July 1, 2016 20:42
@tseaver
Copy link
Contributor

tseaver commented Jul 1, 2016

@bjwatson Ugh, are you saying that the only change we needed was to set is_bundling=True in the CallOptions constructor?

@tseaver tseaver mentioned this pull request Jul 1, 2016
Merged
@bjwatson
Copy link

bjwatson commented Jul 2, 2016

I think so.

@daspecster
Copy link
Contributor Author

@bjwatson, these are the only references to is_bundling in https://github.com/googleapis/gax-python/search?utf8=%E2%9C%93&q=is_bundling&type=Code.

I'm not sure how the discovery stuff works but, what/how would pick up on the is_bundling flag?

@bjwatson
Copy link

bjwatson commented Jul 2, 2016

@daspecster The first reference is the one that does the magic: https://github.com/googleapis/gax-python/blob/75fc5bdbbad46ab1d08d30718ef1485ae211eb5b/google/gax/__init__.py#L134

It depends on bundling settings being configured for the method: https://github.com/googleapis/googleapis/blob/master/google/pubsub/v1/pubsub_gapic.yaml#L227

I'm not sure about Discovery. Are you doing anything with that? It's just generating file-length sample code right now, and not any API surfaces.

@daspecster
Copy link
Contributor Author

@bjwatson, thanks!

I'm not sure on the Discovery question.
@tseaver, do you know?

This was referenced Aug 3, 2016
Closed
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees

@tseaver tseaver

Labels
api: pubsub Issues related to the Pub/Sub API. cla: yes This human has signed the Contributor License Agreement. grpc
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
6 participants
@daspecster @tseaver @bjwatson @geigerj @dhermes @googlebot