Unable to replicate from SG to CouchDB

[ssured]

While pull-replicating from remote SG ADMIN(:4985) to a localhost fresh CouchDB, I get 16 of these SG errors on different ports, before the whole replication halts (without any further feedback) (IP replaced for privacy, if needed I can provide a URL of course)

13:12:14.707014 HTTP: #094: --> 500 Internal error: write tcp IP.IP.IP.IP:63634: broken pipe

13:12:14.712485 WARNING: Couldn't interpret error type *net.OpError, value write tcp IP.IP.IP.IP:63635: broken pipe -- base.ErrorAsHTTPStatus() at error.go:58 2014/01/28 13:12:14 http: multiple response.WriteHeader calls

CouchDB _active_tasks says: (null as progress seems weird)

{
pid: "<0.332.0>",
checkpointed_source_seq: 0,
continuous: false,
doc_id: null,
doc_write_failures: 0,
docs_read: 0,
docs_written: 0,
missing_revisions_found: 0,
progress: null,
replication_id: "79cd81773f26ebeac58fc957cd09223e",
revisions_checked: 1,
source: "http://IP.IP.IP.IP:4985/default/",
source_seq: 74,
started_on: 1390911110,
target: "todo-lite",
type: "replication",
updated_on: 1390911111
}

Issue looks similar to #127

49/62 documents got transferred. Some documents have 100 (png) attachments to them. Total DB size is 52MB

I have CouchBase 2.2 Enterprise installed with the latest Sync Gateway (1.0 beta 2) from the website, installed today.

[snej]

Looks as though CouchDB closed the socket unexpectedly. When this has happened before it's because it throws an internal Erlang exception/crash while handling the HTTP response. Can you check the CouchDB log file for exception logs that happen at the same time as these errors? (They're really hard to interpret, unless you're an Erlang expert who knows the internals of CouchDB, but they can offer some clues about what CouchDB didn't like. In my experience it's pretty brittle about what types of data it accepts.)

[ssured]

I got this from the log (it's from a replication to a couchdb 1.3 instance)

[Tue, 28 Jan 2014 11:46:24 GMT] [error] [<0.3543.15>] {error_report,<0.31.0>,
                       {<0.3543.15>,crash_report,
                        [[{initial_call,
                           {couch_replicator,init,['Argument__1']}},
                          {pid,<0.3543.15>},
                          {registered_name,[]},
                          {error_info,
                           {exit,
                            {worker_died,<0.3567.15>,
                             {process_died,<0.3602.15>,
                              {function_clause,
                               [{couch_replicator_api_wrap,
                                 '-open_doc_revs/6-fun-1-',
                                 [404,
                                  [{"Content-Type","application/json"},
                                   {"Date","Tue, 28 Jan 2014 11:46:04 GMT"},
                                   {"Content-Length","44"}],
                                  #Fun<couch_replicator_httpc.0.14493125>,
                                  <0.3602.15>],
                                 [{file,
                                   "/home/build-couchdb13/dependencies/couchdb/src/couch_replicator/src/couch_replicator_api_wrap.erl"},
                                  {line,171}]},
                                {couch_replicator_httpc,
                                 process_stream_response,5,
                                 [{file,
                                   "/home/build-couchdb13/dependencies/couchdb/src/couch_replicator/src/couch_replicator_httpc.erl"},
                                  {line,111}]},
                                {couch_replicator_api_wrap,
                                 '-open_doc_revs/6-fun-2-',4,
                                 [{file,
                                   "/home/build-couchdb13/dependencies/couchdb/src/couch_replicator/src/couch_replicator_api_wrap.erl"},
                                  {line,166}]}]}}},
                            [{gen_server,terminate,6,
                              [{file,"gen_server.erl"},{line,747}]},
                             {proc_lib,init_p_do_apply,3,
                              [{file,"proc_lib.erl"},{line,227}]}]}},
                          {ancestors,
                           [couch_replicator_job_sup,couch_primary_services,
                            couch_server_sup,<0.32.0>]},
                          {messages,
                           [{'$gen_cast',{report_seq,{7,<<"*:47">>}}},
                            {'$gen_cast',checkpoint},
                            {'EXIT',<0.3564.15>,
                             {process_died,<0.3671.15>,
                              {function_clause,
                               [{couch_replicator_api_wrap,
                                 '-open_doc_revs/6-fun-1-',
                                 [404,
                                  [{"Content-Type","application/json"},
                                   {"Date","Tue, 28 Jan 2014 11:46:14 GMT"},
                                   {"Content-Length","44"}],
                                  #Fun<couch_replicator_httpc.0.14493125>,
                                  <0.3671.15>],
                                 [{file,
                                   "/home/build-couchdb13/dependencies/couchdb/src/couch_replicator/src/couch_replicator_api_wrap.erl"},
                                  {line,171}]},
                                {couch_replicator_httpc,
                                 process_stream_response,5,
                                 [{file,
                                   "/home/build-couchdb13/dependencies/couchdb/src/couch_replicator/src/couch_replicator_httpc.erl"},
                                  {line,111}]},
                                {couch_replicator_api_wrap,
                                 '-open_doc_revs/6-fun-2-',4,
                                 [{file,
                                   "/home/build-couchdb13/dependencies/couchdb/src/couch_replicator/src/couch_replicator_api_wrap.erl"},
                                  {line,166}]}]}}},
                            {'$gen_cast',{report_seq,{8,<<"*:74">>}}},
                            {'$gen_call',
                             {<0.3571.15>,#Ref<0.0.25.52117>},
                             {report_seq_done,
                              {8,<<"*:74">>},
                              {rep_stats,27,27,27,27,0}}},
                            {'EXIT',<0.3556.15>,normal}]},
                          {links,
                           [<0.3559.15>,<0.3560.15>,<0.3570.15>,<0.3557.15>,
                            <0.3558.15>,<0.90.0>]},
                          {dictionary,
                           [{task_status_props,
                             [{checkpointed_source_seq,0},
                              {continuous,true},
                              {doc_id,null},
                              {doc_write_failures,0},
                              {docs_read,5},
                              {docs_written,5},
                              {missing_revisions_found,5},
                              {progress,null},
                              {replication_id,
                               <<"ea59060ef8910d34fe0f22038776a0b3+continuous+create_target">>},
                              {revisions_checked,5},
                              {source,
                               <<"http://IP.IP.IP.:4985/default/">>},
                              {source_seq,74},
                              {started_on,1390909564},
                              {target,<<"todo-lite">>},
                              {type,replication},
                              {updated_on,1390909564}]},
                            {task_status_update,
                             {{1390,909584,482145},1000000}}]},
                          {trap_exit,true},
                          {status,running},
                          {heap_size,6765},
                          {stack_size,24},
                          {reductions,13939}],
                         [{neighbour,
                           [{pid,<0.3561.15>},
                            {registered_name,[]},
                            {initial_call,
                             {ibrowse_http_client,init,['Argument__1']}},
                            {current_function,{gen_server,loop,6}},
                            {ancestors,[<0.3558.15>]},
                            {messages,[]},
                            {links,[<0.3558.15>,#Port<0.7504>]},
                            {dictionary,
                             [{my_trace_flag,false},
                              {ibrowse_trace_token,
                               ["IP.IP.IP.",58,"4985"]},
                              {http_prot_vsn,"HTTP/1.1"},
                              {conn_close,"false"}]},
                            {trap_exit,false},
                            {status,waiting},
                            {heap_size,610},
                            {stack_size,9},
                            {reductions,6711}]},
                          {neighbour,
                           [{pid,<0.3558.15>},
                            {registered_name,[]},
                            {initial_call,{erlang,apply,2}},
                            {current_function,
                             {couch_replicator_httpc,accumulate_messages,3}},
                            {ancestors,[]},
                            {messages,[]},
                            {links,[<0.3543.15>,<0.3561.15>]},
                            {dictionary,
                             [{last_seq,<<"*:74">>},{retries_left,10}]},
                            {trap_exit,false},
                            {status,waiting},
                            {heap_size,2584},
                            {stack_size,34},
                            {reductions,21648}]},
                          {neighbour,
                           [{pid,<0.3557.15>},
                            {registered_name,[]},
                            {initial_call,
                             {couch_work_queue,init,['Argument__1']}},
                            {current_function,{gen_server,loop,6}},
                            {ancestors,
                             [<0.3543.15>,couch_replicator_job_sup,
                              couch_primary_services,couch_server_sup,
                              <0.32.0>]},
                            {messages,[]},
                            {links,[<0.3543.15>]},
                            {dictionary,[]},
                            {trap_exit,false},
                            {status,waiting},
                            {heap_size,1597},
                            {stack_size,9},
                            {reductions,982}]},
                          {neighbour,
                           [{pid,<0.3559.15>},
                            {registered_name,[]},
                            {initial_call,{erlang,apply,2}},
                            {current_function,{gen,do_call,4}},
                            {ancestors,[]},
                            {messages,[]},
                            {links,[<0.3543.15>]},
                            {dictionary,[]},
                            {trap_exit,false},
                            {status,waiting},
                            {heap_size,2584},
                            {stack_size,23},
                            {reductions,375}]}]]}}
[Tue, 28 Jan 2014 11:46:24 GMT] [error] [<0.3560.15>] ** Generic server <0.3560.15> terminating 
** Last message in was {'EXIT',<0.3543.15>,
                        {worker_died,<0.3567.15>,
                         {process_died,<0.3602.15>,
                          {function_clause,
                           [{couch_replicator_api_wrap,
                             '-open_doc_revs/6-fun-1-',
                             [404,
                              [{"Content-Type","application/json"},
                               {"Date","Tue, 28 Jan 2014 11:46:04 GMT"},
                               {"Content-Length","44"}],
                              #Fun<couch_replicator_httpc.0.14493125>,
                              <0.3602.15>],
                             [{file,
                               "/home/build-couchdb13/dependencies/couchdb/src/couch_replicator/src/couch_replicator_api_wrap.erl"},
                              {line,171}]},
                            {couch_replicator_httpc,process_stream_response,
                             5,
                             [{file,
                               "/home/build-couchdb13/dependencies/couchdb/src/couch_replicator/src/couch_replicator_httpc.erl"},
                              {line,111}]},
                            {couch_replicator_api_wrap,
                             '-open_doc_revs/6-fun-2-',4,
                             [{file,
                               "/home/build-couchdb13/dependencies/couchdb/src/couch_replicator/src/couch_replicator_api_wrap.erl"},
                              {line,166}]}]}}}}
** When Server state == {state,<0.3543.15>,<0.3562.15>,20,
                         {httpdb,"http://IP.IP.IP.:4985/default/",nil,
                          [{"Accept","application/json"},
                           {"User-Agent","CouchDB/1.3.0"}],
                          30000,
                          [{socket_options,
                            [{keepalive,true},{nodelay,false}]}],
                          10,250,<0.3544.15>,20},
                         {db,<0.3549.15>,<0.3550.15>,nil,
                          <<"1390909563925564">>,<0.3551.15>,<0.3547.15>,
                          <0.3554.15>,
                          {db_header,6,0,0,nil,nil,nil,0,nil,nil,1000},
                          0,
                          {btree,<0.3547.15>,nil,
                           #Fun<couch_db_updater.10.129614869>,
                           #Fun<couch_db_updater.11.129614869>,
                           #Fun<couch_btree.5.114668118>,
                           #Fun<couch_db_updater.12.129614869>,snappy},
                          {btree,<0.3547.15>,nil,
                           #Fun<couch_db_updater.13.129614869>,
                           #Fun<couch_db_updater.14.129614869>,
                           #Fun<couch_btree.5.114668118>,
                           #Fun<couch_db_updater.15.129614869>,snappy},
                          {btree,<0.3547.15>,nil,
                           #Fun<couch_btree.3.114668118>,
                           #Fun<couch_btree.4.114668118>,
                           #Fun<couch_btree.5.114668118>,nil,snappy},
                          0,<<"todo-lite">>,
                          "/home/build-couchdb13/build/var/lib/couchdb/todo-lite.couch",
                          [],[],nil,
                          {user_ctx,<<"scripts">>,
                           [<<"_admin">>],
                           <<"{couch_httpd_auth, cookie_authentication_handler}">>},
                          nil,1000,
                          [before_header,after_header,on_file_open],
                          [create,
                           {user_ctx,
                            {user_ctx,<<"scripts">>,
                             [<<"_admin">>],
                             <<"{couch_httpd_auth, cookie_authentication_handler}">>}}],
                          snappy,nil,nil},
                         [],nil,nil,nil,
                         {rep_stats,0,0,0,0,0},
                         nil,<0.3563.15>,
                         {batch,[],0}}
** Reason for termination == 
** {worker_died,<0.3567.15>,
       {process_died,<0.3602.15>,
           {function_clause,
               [{couch_replicator_api_wrap,'-open_doc_revs/6-fun-1-',
                    [404,
                     [{"Content-Type","application/json"},
                      {"Date","Tue, 28 Jan 2014 11:46:04 GMT"},
                      {"Content-Length","44"}],
                     #Fun<couch_replicator_httpc.0.14493125>,<0.3602.15>],
                    [{file,
                         "/home/build-couchdb13/dependencies/couchdb/src/couch_replicator/src/couch_replicator_api_wrap.erl"},
                     {line,171}]},
                {couch_replicator_httpc,process_stream_response,5,
                    [{file,
                         "/home/build-couchdb13/dependencies/couchdb/src/couch_replicator/src/couch_replicator_httpc.erl"},
                     {line,111}]},
                {couch_replicator_api_wrap,'-open_doc_revs/6-fun-2-',4,
                    [{file,
                         "/home/build-couchdb13/dependencies/couchdb/src/couch_replicator/src/couch_replicator_api_wrap.erl"},
                     {line,166}]}]}}}

[Tue, 28 Jan 2014 11:46:24 GMT] [error] [<0.90.0>] {error_report,<0.31.0>,
                    {<0.90.0>,supervisor_report,
                     [{supervisor,{local,couch_replicator_job_sup}},
                      {errorContext,child_terminated},
                      {reason,
                       {worker_died,<0.3567.15>,
                        {process_died,<0.3602.15>,
                         {function_clause,
                          [{couch_replicator_api_wrap,
                            '-open_doc_revs/6-fun-1-',
                            [404,
                             [{"Content-Type","application/json"},
                              {"Date","Tue, 28 Jan 2014 11:46:04 GMT"},
                              {"Content-Length","44"}],
                             #Fun<couch_replicator_httpc.0.14493125>,
                             <0.3602.15>],
                            [{file,
                              "/home/build-couchdb13/dependencies/couchdb/src/couch_replicator/src/couch_replicator_api_wrap.erl"},
                             {line,171}]},
                           {couch_replicator_httpc,process_stream_response,5,
                            [{file,
                              "/home/build-couchdb13/dependencies/couchdb/src/couch_replicator/src/couch_replicator_httpc.erl"},
                             {line,111}]},
                           {couch_replicator_api_wrap,
                            '-open_doc_revs/6-fun-2-',4,
                            [{file,
                              "/home/build-couchdb13/dependencies/couchdb/src/couch_replicator/src/couch_replicator_api_wrap.erl"},
                             {line,166}]}]}}}},
                      {offender,
                       [{pid,<0.3543.15>},
                        {name,
                         "ea59060ef8910d34fe0f22038776a0b3+continuous+create_target"},
                        {mfargs,{gen_server,start_link,undefined}},
                        {restart_type,temporary},
                        {shutdown,250},
                        {child_type,worker}]}]}}
[Tue, 28 Jan 2014 11:46:24 GMT] [error] [<0.3560.15>] {error_report,<0.31.0>,
                       {<0.3560.15>,crash_report,
                        [[{initial_call,
                           {couch_replicator_worker,init,['Argument__1']}},
                          {pid,<0.3560.15>},
                          {registered_name,[]},
                          {error_info,
                           {exit,
                            {worker_died,<0.3567.15>,
                             {process_died,<0.3602.15>,
                              {function_clause,
                               [{couch_replicator_api_wrap,
                                 '-open_doc_revs/6-fun-1-',
                                 [404,
                                  [{"Content-Type","application/json"},
                                   {"Date","Tue, 28 Jan 2014 11:46:04 GMT"},
                                   {"Content-Length","44"}],
                                  #Fun<couch_replicator_httpc.0.14493125>,
                                  <0.3602.15>],
                                 [{file,
                                   "/home/build-couchdb13/dependencies/couchdb/src/couch_replicator/src/couch_replicator_api_wrap.erl"},
                                  {line,171}]},
                                {couch_replicator_httpc,
                                 process_stream_response,5,
                                 [{file,
                                   "/home/build-couchdb13/dependencies/couchdb/src/couch_replicator/src/couch_replicator_httpc.erl"},
                                  {line,111}]},
                                {couch_replicator_api_wrap,
                                 '-open_doc_revs/6-fun-2-',4,
                                 [{file,
                                   "/home/build-couchdb13/dependencies/couchdb/src/couch_replicator/src/couch_replicator_api_wrap.erl"},
                                  {line,166}]}]}}},
                            [{gen_server,terminate,6,
                              [{file,"gen_server.erl"},{line,747}]},
                             {proc_lib,init_p_do_apply,3,
                              [{file,"proc_lib.erl"},{line,227}]}]}},
                          {ancestors,
                           [<0.3543.15>,couch_replicator_job_sup,
                            couch_primary_services,couch_server_sup,<0.32.0>]},
                          {messages,[]},
                          {links,[<0.3562.15>,<0.3563.15>]},
                          {dictionary,
                           [{last_stats_report,{1390,909564,400678}}]},
                          {trap_exit,true},
                          {status,running},
                          {heap_size,10946},
                          {stack_size,24},
                          {reductions,1392}],
                         [{neighbour,
                           [{pid,<0.3563.15>},
                            {registered_name,[]},
                            {initial_call,
                             {couch_event_sup,init,['Argument__1']}},
                            {current_function,{gen_server,loop,6}},
                            {ancestors,
                             [<0.3560.15>,<0.3543.15>,
                              couch_replicator_job_sup,couch_primary_services,
                              couch_server_sup,<0.32.0>]},
                            {messages,[{'$gen_cast',stop}]},
                            {links,[<0.3560.15>,<0.88.0>]},
                            {dictionary,[]},
                            {trap_exit,false},
                            {status,runnable},
                            {heap_size,233},
                            {stack_size,9},
                            {reductions,32}]},
                          {neighbour,
                           [{pid,<0.3562.15>},
                            {registered_name,[]},
                            {initial_call,{erlang,apply,2}},
                            {current_function,
                             {couch_replicator_worker,queue_fetch_loop,5}},
                            {ancestors,[]},
                            {messages,[]},
                            {links,[<0.3560.15>]},
                            {dictionary,
                             [{last_stats_report,{1390,909584,482198}}]},
                            {trap_exit,false},
                            {status,waiting},
                            {heap_size,1597},
                            {stack_size,10},
                            {reductions,329}]}]]}}

Quite hard to read indeed ;) Thanks for having a look.

I continued playing with SG today, replication of the todo-lite db from the phonegap example went ok. Does attachment size matter? Does the existence of _design documents matter? I can't think of any other structural difference between the todolite data and my own data.

[snej]

Um, yeah, that didn't tell me much. :( I've sent a shout out to a CouchDB mailing list to see if anyone who knows the source can help interpret it.

Can you identify which HTTP request(s) trigger the exception? For example, in the log line you posted earlier, HTTP: #094: --> 500 Internal error, if you look back a few lines in the log you can see the request line for #094. That would help narrow this down.

[mikerhodes]

Looks like couch is getting a 404 for an open_revs call to me. It'd be good to know the call it was making at the time, and what curling the sync gateway for the same thing gives.

I'm not totally au fait with the replicator code, but guessing from the erlang and the 404 sat in the log.

[ssured]

I did see a 404 on a _design document in sync gateway, I assumed that it was the correct behaviour as sync gateway does not server any underscored files, right?
I'll try if I can narrow down the replication and see if I can get some failing setup.

[snej]

Yes, Sync Gateway doesn't support design documents so it'll return a 404 for those URLs. Could you paste in the exact method/URL of the request(s) that got the broken-pipe errors?

Also, what parameters does the replication use? In particular, is it using a filter?

[ssured]

I'm trying to reproduce the error again. There is a weird situation. From the console running SG I see CouchDB (pulling from SG) makes these initial requests:

22:10:51.488574 HTTP:  #002: HEAD /todos/  (ADMIN)
22:10:51.508740 HTTP:  #003: GET /todos/  (ADMIN)
22:10:51.736942 HTTP:  #004: GET /todos/_local/e0067c23ee314008e418cd3402759a80  (ADMIN)
22:10:51.737104 HTTP:  #004:     --> 404 missing
22:10:51.758577 HTTP:  #005: GET /todos/_local/b25b37a29fc9bc144c7b6e0a42de4d24  (ADMIN)
22:10:51.758653 HTTP:  #005:     --> 404 missing
22:10:51.772953 HTTP:  #006: GET /todos/_local/98d842f6a4209a302841de0a58a36d7a  (ADMIN)
22:10:51.773043 HTTP:  #006:     --> 404 missing
22:10:51.803480 HTTP:  #007: GET /todos/_changes?feed=normal&style=all_docs&since=0&heartbeat=10000  (ADMIN)

Then manually I GET /todos/_changes?feed=normal&style=all_docs&since=0 on :4985 of sync gateway gives

{"results":[
{"seq":"*:10","id":"ba095d73b4ac65f604e295b11e12d502","deleted":true,"changes":[{"rev":"3-f1b2ea03f3a260a88dd4ba2f7687b940"}]}
,{"seq":"*:11","id":"ba095d73b4ac65f604e295b11e12c716","deleted":true,"changes":[{"rev":"2-700aa8992a9fbbaebde473ecd1aca980"}]}
,{"seq":"*:12","id":"ba095d73b4ac65f604e295b11e128c3f","deleted":true,"changes":[{"rev":"5-d05de84b8515fd1b77a36f7169cb0802"}]}
,{"seq":"*:14","id":"_design/e_d","changes":[{"rev":"1-ee841d2f2b768522e51814ff173db4b2"}]}
....

The weird thing here is that the changes feed says _design/e_d lives in the DB. Thus CouchDB will try to fetch the _design/e_d document, but gets a 404.

From SG console:

22:10:52.059931 HTTP:  #010: GET /todos/ba095d73b4ac65f604e295b11e128c3f?revs=true&open_revs=%5B%225-d05de84b8515fd1b77a36f7169cb0802%22%5D&latest=true  (ADMIN)
22:10:52.062540 HTTP:  #011: GET /todos/_design/e_d?revs=true&open_revs=%5B%221-ee841d2f2b768522e51814ff173db4b2%22%5D&latest=true  (ADMIN)
22:10:52.063025 HTTP:  #011:     --> 404 unknown URL
22:10:52.075868 HTTP:  #012: GET /todos/  (ADMIN)
22:10:52.076777 HTTP:  #013: GET /todos/04f0170b9b1029dbb1f33bb6f67c51b1?revs=true&open_revs=%5B%223-7d8a79030c4ba03c924dc52458c277ac%22%5D&latest=true  (ADMIN)
22:10:52.076902 HTTP:  #014: GET /todos/04f0171900111c0b9e908568ca075c41?revs=true&open_revs=%5B%223-4b6b5e00279e0143c06921502181b198%22%5D&latest=true  (ADMIN)
22

And the CouchDB logs give (massive logs!):

[Wed, 29 Jan 2014 21:09:17 GMT] [info] [<0.361.0>] Replication `"e0067c23ee314008e418cd3402759a80+create_target"` is using:
  4 worker processes
  a worker batch size of 500
  20 HTTP connections
  a connection timeout of 30000 milliseconds
  10 retries per request
  socket options are: [{keepalive,true},{nodelay,false}]
[Wed, 29 Jan 2014 21:09:17 GMT] [info] [<0.119.0>] starting new replication `e0067c23ee314008e418cd3402759a80+create_target` at <0.361.0> (`http://IP.IP.IP.IP:4985/todos/` -> `syncg`)
[Wed, 29 Jan 2014 21:09:17 GMT] [error] [emulator] Error in process <0.407.0> with exit value: {function_clause,[{couch_replicator_api_wrap,'-open_doc_revs/6-fun-1-',[404,[{"Content-Type","application/json"},{"Date","Wed, 29 Jan 2014 21:10:52 GMT"},{"Content-Length","44"}],#Fun<couch_replicator_httpc.0.13763845>],[{file,"/Users/jan/Work/build-couchdb-mac... 


[Wed, 29 Jan 2014 21:09:17 GMT] [info] [<0.404.0>] Retrying GET to http://IP.IP.IP.IP:4985/todos/_design/e_d?revs=true&open_revs=%5B%221-ee841d2f2b768522e51814ff173db4b2%22%5D&latest=true in 1.0 seconds due to error {function_clause,
                                                                                                                                                                                            [{couch_replicator_api_wrap,
                                                                                                                                                                                              '-open_doc_revs/6-fun-1-',
                                                                                                                                                                                              [404,
                                                                                                                                                                                               [{"Content-Type",
                                                                                                                                                                                                 "application/json"},
                                                                                                                                                                                                {"Date",
                                                                                                                                                                                                 "Wed, 29 Jan 2014 21:10:52 GMT"},
                                                                                                                                                                                                {"Content-Length",
                                                                                                                                                                                                 "44"}],
                                                                                                                                                                                               #Fun<couch_replicator_httpc.0.13763845>],
                                                                                                                                                                                              [{file,
                                                                                                                                                                                                "/Users/jan/Work/build-couchdb-mac/build-couchdb/git-build/https%3A%2F%2Fgit-wip-us.apache.org%2Frepos%2Fasf%2Fcouchdb.git%3A1.5.x/src/couch_replicator/src/couch_replicator_api_wrap.erl"},
                                                                                                                                                                                               {line,
                                                                                                                                                                                                174}]},
                                                                                                                                                                                             {couch_replicator_httpc,
                                                                                                                                                                                              process_stream_response,
                                                                                                                                                                                              5,
                                                                                                                                                                                              [{file,
                                                                                                                                                                                                "/Users/jan/Work/build-couchdb-mac/build-couchdb/git-build/https%3A%2F%2Fgit-wip-us.apache.org%2Frepos%2Fasf%2Fcouchdb.git%3A1.5.x/src/couch_replicator/src/couch_replicator_httpc.erl"},
                                                                                                                                                                                               {line,
                                                                                                                                                                                                111}]}]}
[Wed, 29 Jan 2014 21:09:17 GMT] [error] [emulator] Error in process <0.432.0> with exit value: {function_clause,[{couch_replicator_api_wrap,'-open_doc_revs/6-fun-1-',[404,[{"Content-Type","application/json"},{"Date","Wed, 29 Jan 2014 21:10:52 GMT"},{"Content-Length","44"}],#Fun<couch_replicator_httpc.0.13763845>],[{file,"/Users/jan/Work/build-couchdb-mac... 

The situation is going on for some time now on my machines, CouchDB is waiting for _design/e_d to not 404. SG is serving 404 for _design/e_d. Thus the replication will never complete.

[ssured]

BTW forgot to tell. Replication is default, initiated as a single replication using futon on CouchDB 1.5

[ssured]

Just as a question, what is the decision behind not serving _design documents through SG? Wouldn't it be logical to just sync them, and treat them like regular documents?

[ssured]

One more question, what is the best way to clear all data from CouchBase. Thus I can clean the setup, push into a clean DB and pull. It might help ruling out causes.

[snej]

Hm, this sounds like it's related to #189. In this case it sounds like someone was able to store a design doc into the database (which I didn't think was possible.) How was that database populated?

We've gone back and forth over the treatment of design docs in the gateway. Currently it rejects design documents, even though it doesn't treat them specially itself, because they're magic to CouchDB and have security implications there since they contain executable code.

[snej]

what is the best way to clear all data from CouchBase

In the bucket settings, enable flushing. Then when you view the bucket there will be a "Flush" button that deletes all docs from the bucket. Make sure you stop the Sync Gateway before doing this, or at least issue a DELETE on that database so it'll stop using it.

[ssured]

I just synced a CouchDB database to SG, but on port 4895, so as admin. My guess is that syncing is admin allowed the design document to enter the db. I'll try again later today.

[janl]

It looks like we are not handling a 404 in the function below, especially the fun(200, Headers, StreamDataFun) -> bit (that’s like 171 in couch_replicator_api_wrap).

open_doc_revs(#httpdb{} = HttpDb, Id, Revs, Options, Fun, Acc) ->
    Path = encode_doc_id(Id),
    QArgs = options_to_query_args(
        HttpDb, Path, [revs, {open_revs, Revs} | Options]),
    Self = self(),
    Streamer = spawn_link(fun() ->
            send_req(
                HttpDb,
                [{path, Path}, {qs, QArgs},
                    {ibrowse_options, [{stream_to, {self(), once}}]},
                    {headers, [{"Accept", "multipart/mixed"}]}],
                fun(200, Headers, StreamDataFun) ->
                    remote_open_doc_revs_streamer_start(Self),
                    {<<"--">>, _, _} = couch_httpd:parse_multipart_request(
                        get_value("Content-Type", Headers),
                        StreamDataFun,
                        fun mp_parse_mixed/1)
                end),
            unlink(Self)
        end),
    receive
    {started_open_doc_revs, Ref} ->
        receive_docs_loop(Streamer, Fun, Id, Revs, Ref, Acc)
    end;

[ssured]

@janl: does that imply that this is an issue of CouchDB?