Streaming API support

[tonimelisma]

The Salesforce streaming API is the best way to hook up to receive near real-time data changes or platform events from processes running on the Salesforce org so I think it would be great to implement support in Salesforce Py.

This is a feature request. I'd welcome all input on whether this is a priority for the product as well as architectural guidance.

At least a couple of open questions:

1. Is there anything re: authentication we need to revamp? I'd expect we can use the SalesforcePy OAuth token after login() to communicate with the Streaming API just as we do with the rest of them?
2. Any ideas on suitable Bayeux libraries? Can we just use https://github.com/SalesforceFoundation/python-bayeux
3. Are there any examples within SalesforcePy I should take mind of that are using something similar (event-driven callbacks) so the implementation would be in-line with the existing code?

[caffalaughrey]

@pepefloyd and I considered this in the early days of the client, perhaps it's time to revive it again. I'll address the questions the best I can, and we can see where this leads.

1. Is there anything re: authentication we need to revamp? I'd expect we can use the SalesforcePy OAuth token after login() to communicate with the Streaming API just as we do with the rest of them?

I agree with your suggestion; I would opt to use what's available. Someday we might want to ask ourselves whether to start using something like requests-oauthlib (if feasible) to a. abstract oauth2 bearer flow for us and b. make the implementation of other oauth2 flows supported by Salesforce simpler. But regardless of how this gets handled, I think a login() call is probably unavoidable.

2. Any ideas on suitable Bayeux libraries? Can we just use https://github.com/SalesforceFoundation/python-bayeux

Whatever we use should be suitably tested, and any new features we develop for our client should also have an easily repeatable pattern for testing too. As for our existing features that make simple requests we've been able to do this effortlessly using the responses package. With CometD however, I don't know what the effort would be. I'm not even sure if they're using http or websocket these days.

But yea regardless, if a package demonstrably meets the suitably tested and easy to test externally requirements, I'm happy to give it my approval.

3. Are there any examples within SalesforcePy I should take mind of that are using something similar (event-driven callbacks) so the implementation would be in-line with the existing code?

A notification-based addition to the client would be a must to support streaming. We've also bandied about the use of this pattern for managing Bulk API 2.0 calls, considering the asynchronous nature of bulk jobs.

So far however, no, there are no examples of notification-based patterns in action. However the examples we have in chatter.py, jobs.py, or wave.py demonstrate the basic requirements for exposing new API services. The guidance I'd give here if you're working on a branch for this is:

a. familiarise yourself with one of the submodules I just mentioned
b. create a streaming namespace and related module file eg. streaming.py
c. expose the primary interface of that module via the Client class in sfdc.py. Whatever said interface is we can define more clearly later, but for now it should probably at least expose a subscribe() function
d. let's see if we can test just a simple subscription to a topic as a 'hello world'.

If that much can be done, I think we can get the ball rolling on a design based on the streaming documentation, as well as determine what logic we can abstract to commons.

[caffalaughrey]

Reviving along with a few other open issues today.

Authentication is coming up in a few places at the moment (#48 and #56 for example). I'll just narrow the scope of this to the streaming part of our discussion here.

@tonimelisma, just had a glance at BayeuxClient and yes, I'd be happy to rely on python_bayeux as the core for interacting with the Streaming API. How to mock SF Bayeux interaction in testing is still an open question for me, however. I note in the foundation code there are tests against local demo services (cometd-demo shows up) but I can't see where it is part of the package. That, as well as the use of getpass leads me to think it must be tested manually. Any ideas how to go about adding it as a dependency and automating a build where we can reliably test our streaming code against it?

[tonimelisma]

Hey @caffalaughrey yeah I think we'd need some sort of Bayeux server. Running the Java-based cometd would require a good deal of trickery on the CI/CD part. Maybe you'd even need to build a Docker image or something like that.. not sure.

I do love contributing to open source. May I inquire why you are waking the project up from slumber? Seems to me like there's not a whole lot of attention during the last few years. Is that about to change? Why is that, is there a need or new demand internally at Salesforce and/or have resources been assigned to work on this?

Personally it's just been a VTO project and I wouldn't mind working on it but it would be nice if there was both active development (supply) as well as more user engagement (demand). Perhaps on the demand side we could also consider some form or marketing..?