CloudFunction connects to an existing subscription and sometimes cannot see messages or sends messages to a topic and they never arrive

[lc-chrisbarton]

A CloudFunction connects to an existing subscription and sometimes cannot see messages while the same code run locally sees messages. This happens using the PubSub library 0.14.5. This issue is intermittent but frequent. Redeploying the function fixes it. There is also a problem adding messages to a topic from a CloudFunction which is also fixed by redeploying the function.

See attached logs of the function connecting to a subscription but seeing no messages while the other log shows the same code run later and locally and seeing messages. The issue is happening now in one function (others are ok). This is a test environment that is just testing this issue so we are controlling messages into the system in order to try to pin down the problem. I will downgrade the client version to 0.13.x and report back in a few days on whether that fixes the issue (issue is intermittent).

cloudfunction.log
local.log

[lc-chrisbarton]

Further investigation shows that the problem only occurs with a cached function and that is why a re-deploy fixes the issue. You can see this in the attached log which shows two executions, one 10 minutes after the other. The first execution is using a cached function and hence all the initialization logs are missing. The second is using an uncached function and hence all the extra initialization logs. In the second case the function sees messages on the subscription which were not seen 10 minutes earlier even though local code showed they were there.

cloudfunction-first-run.log
cloudfunction-second-run.log

[lc-chrisbarton]

I think this is related to this error googleapis/nodejs-spanner#20 i.e. there is a general issue with the node.js client libraries and the caching that occurs in a CloudFunction:

googleapis/nodejs-spanner#20
From lukesneeringer
"If I understand correctly from a skim, essentially the issue is that the client object is being shared between different GCF instances, but since network connections do not work that way (you can not serialize a connection and send it across the wire), you end up in a wonky state.
The fix is to either not permit client objects to be serialized and shared (like what Python does) or to ensure that the connection object is dropped on serialization and able to be reinitialized on reload."

[lc-chrisbarton]

Oh and that issue is not 10 days old, it is a p1 bug from Jul 10th :(
-> Original issue #2489

[stephenplusplus]

Thank you for the extensive reports. I'm going to close this and let #2438 be the source of truth for this issue. As it stands, it seems like a problem within GCF that we can't fix from our library. We have a small list of workarounds that helped some users.

If @lukesneeringer or anyone else sees a way for us to resolve this from our codebase, please share in #2438.

[lc-chrisbarton]

@stephenplusplus I have pinned to version 0.14.0 of the PubSub client library and this issue does not occur so somewhere between that version and 0.14.5 CloudFunction support has broken. BTW issue #2438/#2489 are > 3 months old and PubSub was working a month ago so this may or may not be related.

[stephenplusplus]

Locking the issue just to make sure we funnel all useful info to one place (#2438).