GetValueAsync<> always failed with "unable to locate node with key"

[ZeekoZhu]

GetValueAsync<> always failed with unable to locate node with key under high concurrency.

A simple reproduce: https://github.com/ZeekoZhu/memcachedcore-stress

Here is the output on my computer:

Use GetValueAsync
Single Run: 1/1/01 8:05:00 AM
Single Run: 1/1/01 8:05:00 AM
Time: 21458ms
Error Cnt: 999988
Avg: 0.021458ms
Use Get
Single Run: 1/1/01 8:05:00 AM
Single Run: 1/1/01 8:05:00 AM
Time: 25200ms
Error Cnt: 0
Avg: 0.0252ms

Get<> is much more reliable than GetValueAsync<>

[cnblogs-dudu]

I can't reproduce it on the server.

Use GetValueAsync
Single Run: 1/1/0001 8:00:00 AM
Single Run: 1/1/0001 8:00:00 AM
Time: 36939ms
Error Cnt: 0
Avg: 0.036939ms
Use Get
Single Run: 1/1/0001 8:00:00 AM
Single Run: 1/1/0001 8:00:00 AM
Time: 41387ms
Error Cnt: 0
Avg: 0.041387ms

[ZeekoZhu]

@cnblogs-dudu Try this test: https://github.com/timy07/EnyimMemcachedCoreTest

[cnblogs-dudu]

GetAsync performs as well as Get in the EnyimMemcachedCoreTest.

[deferz]

I test it on the dev.

• EnyimMemcachedCore 2.1.13

sudo siege -c 1000 -t 10s http://localhost:5000/api/values/async
** SIEGE 3.0.8
** Preparing 1000 concurrent users for battle.
The server is now under siege...
Lifting the server siege...      done.

Transactions:                    428 hits
Availability:                  36.58 %
Elapsed time:                   9.09 secs
Data transferred:               0.02 MB
Response time:                  7.72 secs
Transaction rate:              47.08 trans/sec
Throughput:                     0.00 MB/sec
Concurrency:                  363.42
Successful transactions:         428
Failed transactions:             742
Longest transaction:            8.84
Shortest transaction:           0.23

• EnyimMemcachedCore 2.1.10

sudo siege -c 1000 -t 10s http://localhost:5000/api/values/async
** SIEGE 3.0.8
** Preparing 1000 concurrent users for battle.
The server is now under siege...
Lifting the server siege...      done.

Transactions:                    373 hits
Availability:                  39.10 %
Elapsed time:                   9.96 secs
Data transferred:               0.02 MB
Response time:                  8.01 secs
Transaction rate:              37.45 trans/sec
Throughput:                     0.00 MB/sec
Concurrency:                  299.97
Successful transactions:         373
Failed transactions:             581
Longest transaction:            8.83
Shortest transaction:           0.00

[cnblogs-dudu]

It has been reproduced on Linux. It can‘t be reproduced on Windows.

[cnblogs-dudu]

@ZeekoZhu @timy07 Try EnyimMemcachedCore 2.2.0-beta2.

[deferz]

@cnblogs-dudu It's like the same as before.

• EnyimMemcachedCore 2.2.0-beta2
• sudo siege -c 1000 -t 10s http://localhost:5000/api/values/async

Transactions:                    440 hits
Availability:                  36.12 %
Elapsed time:                   9.81 secs
Data transferred:               0.03 MB
Response time:                  8.06 secs
Transaction rate:              44.85 trans/sec
Throughput:                     0.00 MB/sec
Concurrency:                  361.56
Successful transactions:         440
Failed transactions:             778
Longest transaction:            9.66
Shortest transaction:           0.00

• Exception

fail: Enyim.Caching.MemcachedClient[0]
      GetAsync("TEST_CACHE_KEY")
System.Threading.SemaphoreFullException: Adding the specified count to the semaphore would cause it to exceed its maximum count.
   at System.Threading.Semaphore.Release(Int32 releaseCount)
   at Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl.ReleaseSocket(PooledSocket socket)
   at Enyim.Caching.Memcached.MemcachedNode.ExecuteOperationAsync(IOperation op)
   at Enyim.Caching.Memcached.MemcachedNode.Enyim.Caching.Memcached.IMemcachedNode.ExecuteAsync(IOperation op)
   at Enyim.Caching.MemcachedClient.GetAsync[T](String key)
fail: Enyim.Caching.MemcachedClient[0]
      GetAsync("TEST_CACHE_KEY")
System.Threading.SemaphoreFullException: Adding the specified count to the semaphore would cause it to exceed its maximum count.
   at System.Threading.Semaphore.Release(Int32 releaseCount)
   at Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl.ReleaseSocket(PooledSocket socket)
   at Enyim.Caching.Memcached.MemcachedNode.ExecuteOperationAsync(IOperation op)
   at Enyim.Caching.Memcached.MemcachedNode.Enyim.Caching.Memcached.IMemcachedNode.ExecuteAsync(IOperation op)
   at Enyim.Caching.MemcachedClient.GetAsync[T](String key)
fail: Enyim.Caching.MemcachedClient[0]
      GetAsync("TEST_CACHE_KEY")
System.Threading.SemaphoreFullException: Adding the specified count to the semaphore would cause it to exceed its maximum count.
   at System.Threading.Semaphore.Release(Int32 releaseCount)
   at Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl.ReleaseSocket(PooledSocket socket)
   at Enyim.Caching.Memcached.MemcachedNode.ExecuteOperationAsync(IOperation op)
   at Enyim.Caching.Memcached.MemcachedNode.Enyim.Caching.Memcached.IMemcachedNode.ExecuteAsync(IOperation op)
   at Enyim.Caching.MemcachedClient.GetAsync[T](String key)
fail: Enyim.Caching.MemcachedClient[0]

[cnblogs-dudu]

This issue occurs in the following code in MemcachedNode of EnyimMemcachedCore.

if (!this.semaphore.WaitOne(this.queueTimeout))
{
    message = "Pool is full, timeouting. " + _endPoint;
    if (_isDebugEnabled) _logger.LogDebug(message);
    result.Fail(message, new TimeoutException());

    // everyone is so busy
    return result;
}

[cnblogs-dudu]

This issue disappears when changing inputStream.ReadAsync to inputStream.Read in PooledSocket.ReadAsync

[cnblogs-dudu]

This issue has been fixed in 2.2.0-beta5.