Don't add heartbeat to non-live changes requests #6089

Closed
anowak opened this Issue Dec 30, 2016 · 1 comment

Projects

None yet

2 participants

@anowak
Contributor
anowak commented Dec 30, 2016

At the moment, HTTP adapter adds heartbeat=10000 to any changes request, if heartbeat option is not specified. However, according to CouchDB documentation, this parameter only makes sense for longpoll, continuous and eventsource feeds.

On its own, this is rather harmless. The only real downside, beside sending a couple of additional bytes, is making it harder to distinguish between live and 'non-live' requests by URL.

Our specific use case for changing this is rather convoluted, but I think it may be worth documenting (although there is a better place than this issue for sure), as it may turn useful to other PouchDB users.

Recently we started to observe that some clients try to open enormous number of longpoll change feeds (in hundreds, way beyond the typical browser limit of 6 connections for one domain). After investigation, it turned out that these users are behind web security appliances / gateways, for example Cisco WSA or Zscaler. These components are capable of 'SSL decryption' (basically, a MITM attack with forged SSL certificates). They seem to open longpoll connections to our server on behalf of PouchDB, but are handling heartbeats incorrectly. Basically, they expose similar behavior as a transparent proxy that buffers change feed response:

  1. PouchDB initiates request _changes with feed=longpoll&heartbeat=10000&timeout=25000&since=N.
  2. Web gateway intercepts this request and proxies it.
  3. If there are no changes for 30 seconds, CouchDB sends 3 heartbeats, but they are buffered by the web gateway, so no data ever reaches Pouch.
  4. Connection initiated by PouchDB (to the gateway) times out after 30s, but the gateway still happily keeps the connection to CouchDB open, because data is still flowing. On the other hand, it has no way of figuring that the connection to PouchDB is half-closed.
  5. PouchDB notices that the connection timed out, so we start a new one (get back to the point 1). After couple of minutes, we have dozens of dangling connections from the web gateway to the CouchDB server.

Our solution to this problem is limiting number of open connections per user to 6, using nginx's http_limit_conn_module. When the limit is hit, we respond with 429 (Too Many Requests). After receiving this status code in the client, we globally switch off heartbeats on the replications. To let these requests through (as we don't have an automatic way of killing previous connections), we rely on the value of the heartbeat query param.

So, because one-shot changes request also use heartbeat=10000, they fall into the limit. We can work it around by using some magic number like 10001 for 'legal' continuous changes requests, but I think it is better to disable this default, as it doesn't apply anyways.

@anowak anowak added a commit to evidenceprime/pouchdb that referenced this issue Dec 30, 2016
@anowak anowak (#6089) - Don't add default heartbeat to non-live `_changes` requests
Per [CouchDB documentation](http://docs.couchdb.org/en/2.0.0/api/database/changes.html)
`heartbeat` is not applicable if `feed` isn't `longpoll`, `continuous`
or `eventsource`.
370467d
@anowak anowak added a commit to evidenceprime/pouchdb that referenced this issue Dec 30, 2016
@anowak anowak (#6089) - Don't add default heartbeat to non-live `_changes` requests
Per [CouchDB documentation](http://docs.couchdb.org/en/2.0.0/api/database/changes.html)
`heartbeat` is not applicable if `feed` isn't `longpoll`, `continuous`
or `eventsource`.
c8aa518
@daleharvey daleharvey added a commit that referenced this issue Jan 1, 2017
@anowak @daleharvey anowak + daleharvey (#6089) - Don't add default heartbeat to non-live `_changes` requests
Per [CouchDB documentation](http://docs.couchdb.org/en/2.0.0/api/database/changes.html)
`heartbeat` is not applicable if `feed` isn't `longpoll`, `continuous`
or `eventsource`.
5b692a1
@anowak anowak self-assigned this Jan 1, 2017
@daleharvey
Member

Fixed in 5b692a1

@daleharvey daleharvey closed this Jan 1, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment