New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Network performance issue (30 secs for 8mb PUT) #1409

Closed
stevedrew opened this Issue Jun 23, 2018 · 10 comments

Comments

Projects
None yet
3 participants
@stevedrew

stevedrew commented Jun 23, 2018

Expected Behavior

time curl -N -X PUT http://admin:pass@somserver.com:9999/test/doc2001/file.txt -H "Content-Type: application/octet-stream" -d@8mb_file.dat

Should complete in a reasonable amount of time, ie < 1 second (its only a 8mb file).

Current Behavior

The above command if issued from a remote client (but over a very fast network) can take up to 30 seconds.

The same command issue from a client on the same network takes < 1 second.

Network is very fast between servers, an scp of same file takes < 0.1 seconds.

Tested on dedicated servers and multiple AWS servers. Always same result.

It seems like maybe couchdb network layer maybe sending too small a packet?

I have configured a total of 6 servers mix of AWS and dedicated, all experience the same slowness

Steps to Reproduce (for bugs)

  1. Setup a 16.04 Ubuntu server
  2. Installed couchdb per documentation
  3. Issue a PUT of a 8MB binary file from a remote server to couchdb server

Your Environment

  • Version used: 2.1.1
  • Browser Name and version: curl
  • Operating System and version: Ubuntu 14.04 or 16.04

just single nodes and no special configuration.

@wohali

This comment has been minimized.

Member

wohali commented Jun 23, 2018

Attachments are a known weak point of CouchDB. What happens if you attempt to PUT an 8MB JSON doc (not an attachment, just an 8MB document)?

@stevedrew

This comment has been minimized.

stevedrew commented Jun 23, 2018

I base64 encoded the file as a text file then did the PUT with application/text and it was worse 1 minute 29 seconds.

So this is a 10MB text file that now takes 1 min 29 seconds to PUT.

@stevedrew

This comment has been minimized.

stevedrew commented Jun 23, 2018

And I don't believe this issue is todo with attachments as the same PUT that takes 1 min 29 secs takes less than 1 second if it is done from a client on the same local network as the couchdb server.

Must be some network packet size/timing issue in the network layer of couchdb.

@stevedrew

This comment has been minimized.

stevedrew commented Jun 24, 2018

OK good news found solution to performance issue. Turns out the recbuf setting is the cause.

I had to comment out the recbuf param in mochiweb/mochiweb_socket_server.erl and then the time to put a 10 MB text file went from 1 min 29 seconds to < 1 second !!

Could this be the cause of all the complaints about how slow couchdb is for attachments?

I believe not setting recbuf on linux then allows the operating system to handle it more efficiently. (When set it seems to be causing some tcp window size performance issues that lead to the huge delay for larger PUTs.)

I also found this PR mochi/mochiweb#153 to fix mochiweb to allow an "undefined" setting for recbuf, but couchdb 2.1.1 is not using this version, I tried using this version with couchdb but couchdb did not seem to allow undefined as a param to recbuf in the ini file.

So it would seem like we need couchdb to support setting recbuf to undefined, maybe even having that a the default?

Thanks,
Steve.

@wohali

This comment has been minimized.

Member

wohali commented Jun 25, 2018

/cc @rnewson @davisp @kocolosk not sure when we changed this originally...is the recbuf tweak still required on modern Linuxes?

@kocolosk

This comment has been minimized.

Member

kocolosk commented Jun 25, 2018

I think this one dates back to https://issues.apache.org/jira/browse/COUCHDB-1986. I wasn't too involved with that investigation but when I read through the comments it seems that we didn't have a great reason for customizing the buffer size beyond the fact that it improved things in those specific scenarios where tests were timing out. Note that under the hood mochi was using a fixed recbuf as well; we were just bumping up that fixed value.

CouchDB master has upgraded the mochiweb dependency to v2.17.0 which includes the option to set the buffer to undefined and let the OS manage it. Probably a good idea to start testing with the buffer set to undefined now so we can evaluate whether we want to switch to that as a default. I'm with @stevedrew that we probably do want to go that route.

@wohali

This comment has been minimized.

Member

wohali commented Jun 25, 2018

@kocolosk thank you for the sleuthing and memory here! I'll get a PR up to change the default in master to undefined. That should resolve this issue.

@stevedrew for reference, I help people run CouchDB on AWS all the time, and we've not seen this one crop up before - presumably because people are on low-latency links to AWS from their clients/app servers, whereas mochi/mochiweb#153 points the finger at high-latency links. So, thank you for the report!

wohali added a commit that referenced this issue Jun 25, 2018

@wohali wohali added enhancement and removed waiting on user labels Jun 25, 2018

@wohali

This comment has been minimized.

Member

wohali commented Jun 25, 2018

@stevedrew this isn't the root cause for CouchDB's slow attachment behaviour universally, no. A lot of that slowness comes from the serialisation of attachment data both into the b-tree and over the wire. Internal attachment replication between nodes in a 2.x cluster is also unoptimized, and can block other operations on very large files, leading to database-wide issues.

We still recommend keeping attachments in CouchDB below 10MB per document, which you have already met, hooray! There is an ongoing discussion about how to help guide users towards these defaults, see #1200 and #1253.

@stevedrew

This comment has been minimized.

stevedrew commented Jun 25, 2018

gotcha thanks. Yes in fact our attachments are actually less than 1MB, but we just noticed very slow network transfers even with that, and then tested with 8MB to pull out the problem.

wohali added a commit that referenced this issue Jun 25, 2018

wohali added a commit that referenced this issue Jun 25, 2018

@wohali

This comment has been minimized.

Member

wohali commented Jul 23, 2018

Due to the concerns about changing the defaults here, and the need to get a 2.2.0 release out the door, the change to CouchDB's defaults will not happen until post-2.2.0.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment