revisions are missing on sync-gateway with large number of docs

[sridevi-15]

Library Version

2.5.0-184

.NET Runtime

Xamarin Android
Xamarin-iOS

Operating System / Device Details

ios simulator, Android emulator

Log Output

https://gist.github.com/sridevi-15/47432692153057553e67730a4dcf4012

Steps To Reproduce

1. Create 2 DBs in 2 SGS and Create docs in CBL
2. Set up 1 SG with revs limit 30 and SG2 with revs limit 25.
3. Do push replication to two SGS(each DB
   to each SG) with continuous
4. Do updates on CBL for 35 times.
5. Continue push replication to SGs

Actual behavior

Few docs are having 7 revisions , some have 2 , some have 28. Some have all 30 revisions on first sgw db, same for 2nd sgw db too.

Expected behavior:

All Revisions which updated on CBL should get pushed to SGW for all docs . Should see 30 revisions on both sgw dbs

Reproduction Project

pytest -s -rsx --timeout 1800 --liteserv-version=2.5.0-184 --liteserv-host=192.168.33.109 --liteserv-port=8080 --xattrs --sync-gateway-version=2.5.0-239 --mode=cc --server-version=5.5.0-2473 --liteserv-platform=xamarin-android --create-db-per-test=cbl-test testsuites/CBLTester/topology_specific_tests/multiple_sync_gateways -k test_multiple_sgs_with_differrent_revs_limit[listener_tests/multiple_sync_gateways-1000]

[borrrden]

If this is marked as a regression, please mark the latest version that passed.

[borrrden]

I am having trouble running the 1000 item test. I am not using an emulator, but a Nexus 5 device that I have. 100 (and even 500) passes every time but 1000 crashes with absolutely no information (no stack trace, no local variables, no tombstone, nothing but a SIGSEGV or sometimes SIGBUS). My guess is that the repeated large HTTP requests are getting it in trouble.

I saw no signs of the messages mentioned in this report, but it should be mentioned that I used a manual environment setup. Is this reliant on the server being 5.5.0 ? Also is the following config OK?

Machine 1:

{
    "adminInterface":":4985",
    "verbose":"true",
    "logging": {
        "log_file_path": "/Users/borrrden",
        "error": { "enabled": true },
        "warn": { "enabled": true },
        "info": { "enabled": true },
        "debug": { "enabled": true }
    },
    "databases":{
        "sg_db1": {
            "enable_shared_bucket_access": true,
            "server":"http://localhost:8091",
            "bucket":"test",
            "username": "Administrator",
            "password": "password",
            "num_index_replicas": 0,
            "revs_limit": 30        
        }
    }
}

Machine 2:

{
    "adminInterface":":4985",
    "verbose":"true",
    "logging": {
        "log_file_path": "C:\\Development\\logs",
        "error": { "enabled": true },
        "warn": { "enabled": true },
        "info": { "enabled": true },
        "debug": { "enabled": true }
    },
    "databases":{
        "sg_db2": {
            "enable_shared_bucket_access": true,
            "server":"http://localhost:8091",
            "bucket":"test2",
            "username": "Administrator",
            "password": "password",
            "num_index_replicas": 0,
            "revs_limit": 25   
        }
    }
}

[borrrden]

Test passes with 1000 documents in 2.1.2 but crashes in 2.5.0-b0184 so the crash does look like a legitimate regression of some type. I'll try to get with @bmeike and see if he has any bright ideas about coaxing more information out of signals with Android.

[sridevi-15]

@borrrden , I got slow performed laptop from IT due to issues with previous laptop. Due to that, test was consistently failing when I run with xamarin-ios and xamarin-android. I have asked @euniceatcouchbase to run tests multiple times with different parameters both on xamarin-ios and xamarin-android. She ran with different combinations . She saw most of the times it fails, but once in 4 times passes. She added sleep time and was able to get better pass rate. So far, me or Eunice did not see the crash. It does not look like a bug. It is a bug on my laptop which I am waiting for IT to replace it.
I can close it if you are okay. If there a something related to crash, we can have another ticket

[borrrden]

Ok, let's close it then. I was able to get the test to pass with 1000 for b184 by reducing the number of documents that the test server sends at once (i.e. if it is going to update more than 500, it will split it into batches of 500 so there will be two HTTP calls each with 500 documents in the case of 1000).

I don't have the necessary tools to profile Xamarin Android (they are expensive) but I noticed a significant memory usage reduction when limiting the number of documents send to the update documents call. I tried batches of 100 at first and the native memory usage dropped to 50% of 1000 and the managed memory use dropped to about 10%. My only guess is that .NET is using up a lot of memory to deserialize these giant JSON strings and then when something in LiteCore finally goes over the memory limit it gets a signal. I wish there were more information than that but it could be my device. It is getting pretty old and is only upgradable to Android 6.0.1. Anyway it seems you are satisfied so I will close this ticket.