Document with attachment not being passed to sync function

[ghost]

I have seen this working but now when my iOS device syncs with sync_gateway the documents with attachments are not passed to the sync function, here is a console output from sync_gateway

2013/03/07 11:29:24 GET /sync_gateway/_local/e1409ef09f09f0b88dd299f0d99e1416fb4230e2
2013/03/07 11:29:24 GET /sync_gateway/_local/c873146004ce79ba6bc2885df5c6fd54f5b15665
2013/03/07 11:29:24 GET /sync_gateway/_changes?feed=normal&heartbeat=300000&style=all_docs&limit=100&filter=sync_gateway/bychannel&channels=*
2013/03/07 11:29:24 GET /sync_gateway/_changes?feed=longpoll&heartbeat=300000&style=all_docs&limit=100&filter=sync_gateway/bychannel&channels=*
2013/03/07 11:29:24 waiting for a revision...
2013/03/07 11:29:36 POST /sync_gateway/_revs_diff
2013/03/07 11:29:36 POST /sync_gateway/_bulk_docs
2013/03/07 11:29:36 POST /sync_gateway/_revs_diff
2013/03/07 11:29:36 JS: first if evaluated false
2013/03/07 11:29:36 JS: second if evaluated false
2013/03/07 11:29:36 Doc "E6170221157F793648E799025D7413DFEEB50A6E" grants access: map[]
2013/03/07 11:29:36 ...done waiting
2013/03/07 11:29:36 JS: first if evaluated true
2013/03/07 11:29:36 JS: second if evaluated false
2013/03/07 11:29:36 Doc "317585690691231914726A978C987EF45F0E55D5" grants access: map[]
2013/03/07 11:29:36 waiting for a revision...
2013/03/07 11:29:37 PUT /sync_gateway/DF40608948B59C24E3387646325AE8F312BB9B4F?new_edits=false
2013/03/07 11:29:39 ...done waiting
2013/03/07 11:29:39 GET /sync_gateway/_changes?feed=longpoll&heartbeat=300000&style=all_docs&since=2&limit=100&filter=sync_gateway/bychannel&channels=*
2013/03/07 11:29:39 waiting for a revision...
2013/03/07 11:29:42 PUT /sync_gateway/_local/c873146004ce79ba6bc2885df5c6fd54f5b15665

2013/03/07 11:29:45 PUT /sync_gateway/_local/e1409ef09f09f0b88dd299f0d99e1416fb4230e2

I am using a dummy sync function that does no mapping but logs whether tests for the existence of doc properties evaluate to true or false.

Two documents without attachments look like they get sync'd with POST /sync_gateway/_bulk_docs

And you can see the sync function being called

But the document that has an attachment looks like it gets sync'd with

PUT /sync_gateway/DF40608948B59C24E3387646325AE8F312BB9B4F?new_edits=false

And the sync function is not called.

The attachments property of the failing document is below:

"_attachments": {
"data": {
"content_type": "application/octet-stream",
"digest": "sha1-30BgiUi1nCTjOHZGMlro8xK7m08=",
"length": 1747625,
"revpos": 1,
"stub": true
}
}

[snej]

This is weird. TouchDB pushes docs with attachments differently, using a multipart PUT rather than grouping them into a _bulk_docs POST, so that it can avoid base64-encoding the attachments. So they are going through a different code path in the gateway server. But those code paths are only different in the rest. package — by the time they get to the database-level code they're going into the same method, Database.PutExistingRev.

Unfortunately the sync gateway doesn't yet have the same level of conditional logging that Couchbase Lite does, so I can't just ask you to turn on some logging flags. I'll add some logging to my copy, then send you a patch, if you'd be OK with adding that?

[snej]

Another thought: Are you sure the doc actually got added to the database? Maybe it was rejected for some reason. Running the gateway with the "-v" flag will add logging for failed requests, so you'd be able to see the status code that got returned if it's not 200. TouchDB would also log a warning if a doc it tried to sync got rejected.

[ghost]

Jens

Hi, yes sure, this is a blocker for me so anything to help, I can work around the sync function issues.

Yes, the doc made it to the database, that's where I cut the attachments info from.

and there is a matching attachment document

_sync:att:sha1-30BgiUi1nCTjOHZGMlro8xK7m08=

Andy

[ghost]

Jens

Hi, please hold on this, I just backed off to a sync function that just logs a message and does no eval, this got called for the file with attachment and I also saw th grants access: map[] message for that file, which I didn't see in the original console output. So it looks like this is the eval failing again without generating any kind of error.

Let me dig deeper and I'll report back.

Andy

[ghost]

Jens

Apologies, I had a second level property eval in the first if statement too, the first level property exists in all files without attachments but not in ones with attachments.

So this is a non issue, the issue is the lack of error reporting in otto, which they acknowledge on their project page.

Andy

[snej]

You're saying that the sync function threw a JS exception but it didn't get reported as an error to the Go caller? Hm, that's bad. I think I can work around it, though.

[ghost]

Jens

Hi, well I'm guessing that the exception is thrown internally but there is no manifestation of it externally, except that the if() statement does not evaluate to either true or false. I think that the very first stack trace I reported in the sync function and that you patched so that it was not thrown was probably due to this issue as it was complaining about a Null value.

As a test to see what the expected behaviour should be, I entered the following script into Firefox scratchpad

doc = {
"_attachments": {
"data": {
"content_type": "application/octet-stream",
"digest": "sha1-30BgiUi1nCTjOHZGMlro8xK7m08=",
"length": 1747625,
"revpos": 1,
"stub": true
}
}
}

if(doc._attachments.data.digest) { console.log("digest evaluated true"); } else { console.log("digest evaluated false") }
if(doc._attachments.noexist.digest) { console.log("noexist evaluated true"); } else { console.log("noexist evaluated false") }

The first test evaluates true and writes "digest evaluated true" to the web console as expected.

The second test throws the following exception in the scratchpad window and writes nothing to the console.log.

/*
Exception: doc._attachments.noexist is undefined
@Scratchpad/1:23
*/

So it looks like the otto interpreter is acting correctly in not evaluating to either true or false, but no exception is thrown so there is no way to handle these cases and docs get added without the intended channel mapping.

Andy

[ghost]

Jens

In addition I can put a try catch around the second test and trap the error:

try
{
if(doc._attachments.noexist.digest) { console.log("noexist evaluated true"); } else { console.log("noexist evaluated false") }
}
catch(err)
{
console.log("caught an error: "+err);
}

So I'm wondering if the exception is being swallowed by the sync function as though an explicit throw() had been called as in the sample sync function for the chat demo App?

Andy

[ghost]

Jens

I think I have proved what I said previously, if I use the following sync function:

function(doc, oldDoc, userCtx, secObj) { throw("I dont want to sync this file"); }

Then in the console no documents are mapped, but also I don't see the throw message in the console.

2013/03/07 23:43:21 GET /sync_gateway/_local/c873146004ce79ba6bc2885df5c6fd54f5b15665
2013/03/07 23:43:21 GET /sync_gateway/_local/e1409ef09f09f0b88dd299f0d99e1416fb4230e2
2013/03/07 23:43:21 POST /sync_gateway/_revs_diff
2013/03/07 23:43:21 PUT /sync_gateway/DF40608948B59C24E3387646325AE8F312BB9B4F?new_edits=false
2013/03/07 23:43:21 POST /sync_gateway/_bulk_docs
2013/03/07 23:43:21 GET /sync_gateway/_changes?feed=normal&heartbeat=300000&style=all_docs&limit=100&filter=sync_gateway/bychannel&channels=*
2013/03/07 23:43:21 GET /sync_gateway/_changes?feed=longpoll&heartbeat=300000&style=all_docs&since=2&limit=100&filter=sync_gateway/bychannel&channels=*
2013/03/07 23:43:21 waiting for a revision...
2013/03/07 23:43:23 ...done waiting
2013/03/07 23:43:23 waiting for a revision...
2013/03/07 23:43:26 PUT /sync_gateway/_local/c873146004ce79ba6bc2885df5c6fd54f5b15665
2013/03/07 23:43:27 PUT /sync_gateway/_local/e1409ef09f09f0b88dd299f0d99e1416fb4230e2

So the actual issue is to indicate that the sync function threw an error and print the error message.

I'll close this issue and raise one for the actual issue.

Andy