What does Environment maxreaders reached (-30790) actually mean?

[davidwynter]

I successfully populated the Dbi with the data I am now trying to get. But when I try to get it I get this error, which does not give me enough information to understand what the problem is, as the term "reader" is not mentioned in the TestTutorial.java code even once.

Caused by: org.lmdbjava.Env$ReadersFullException: Environment maxreaders reached (-30790)
at org.lmdbjava.ResultCodeMapper.checkRc(ResultCodeMapper.java:98)
at org.lmdbjava.Txn.<init>(Txn.java:67)
at org.lmdbjava.Env.txn(Env.java:361)
at org.lmdbjava.Env.txnRead(Env.java:370)
at uk.co.example.LmDbStore.getLatestDemoQCBBlock(LmDbStore.java:564)

Here is the env setup code:

 Dbi<ByteBuffer> demoenv = create()
            .setMapSize(20971520l)
            .setMaxDbs(5)
            .open(new File(csEnvironmentStr+"/csdbdemo"));
	demoQcbStore = demoenv.openDbi("demoQcbStoreLocn", MDB_CREATE);

Here is how I store the qcbblock:

final ByteBuffer key = allocateDirect(40);
    ByteBuffer val = null;
	try {
		val = toByteArray(qcb);
	} catch (IOException e) {
		logger.error("", e);			
	}

    try (Txn<ByteBuffer> txn = demoenv.txnWrite()) {    	
        key.put(id.getBytes(UTF_8)).flip();
        demoQcbStore.put(txn, key, val);
        // An explicit commit is required, otherwise Txn.close() rolls it back.
        txn.commit();
    }
...
// toByteArray and toObject are taken from: http://tinyurl.com/69h8l7x with mods for ByteBuffer
public static ByteBuffer toByteArray(Object obj) throws IOException {
    byte[] bytes = null;
    ByteArrayOutputStream bos = null;
    ObjectOutputStream oos = null;
    try {
        bos = new ByteArrayOutputStream();
        oos = new ObjectOutputStream(bos);
        oos.writeObject(obj);
        oos.flush();
        bytes = bos.toByteArray();
    } finally {
        if (oos != null) {
            oos.close();
        }
        if (bos != null) {
            bos.close();
        }
    }
    final ByteBuffer value = allocateDirect(bytes.length);
    value.put(bytes).flip();
    return  value;
}

Here is the getLatestDemoQCBBlock code, last line is where the exception is thrown:

 public QCBBlock getLatestDemoQCBBlock(String id) {
    final ByteBuffer key = allocateDirect(40);
    key.put(id.getBytes(UTF_8)).flip();
    QCBBlock block = null;
    try (Txn<ByteBuffer> txn = demoenv.txnRead()) {

Any explanation?

[krisskross]

A reader is a thread. LMDB has a lock table that track readers/threads in the environment. The default value is 126. If the MDB_NOTLS flag is used, each reader/thread is instead tied to the transaction object until it or the environment object is destroyed. You can read more about this in the LMDB docs.

Your code looks correct from what I can see. Do you have multiple threads that calls getLatestDemoQCBBlock in parallel? Or some other code that might not close their transactions properly? If your transactions are not closed properly the thread will occupy a slot in the lock table, which eventually will exceed the limit which cause this exception.

[krisskross]

Did you find an answer to the exception being thrown?

[chriswyatt]

I had the same problem, and fixed it by using setMaxReaders when creating the environment. My implementation would have only been creating 2 readers at most, so I'm not sure why I was getting this error. I set the value to 8, which is reasonable for my needs right now. I haven't tested my code yet much though, so I can't be sure it wasn't just a bug in my code.

I'm using OpenJDK 1.8 on a Debian Jessy machine.

[davidwynter]

I had not been closing the transactions in some cases, on long test runs it simply ran out of readers.

[krisskross]

Transactions are a bit tricky. I always keep transactions (tx) private to each thread, which means that the thread that started the tx is responsible for closing it. Closing a tx happens as soon as you can answer the query asked for. In a web application this is usually right before a HTTP response is written back to the client. Therefore the lifetime of a tx is usually measured in milliseconds not seconds.

If you have a web server fronting LMDB [1], make sure you know how many "service threads" are handling requests. A "service thread" is different from an IO thread (which usually just shuffle incoming network socket bytes asynchronous to a "service thread").

So be aware of the number of concurrent threads that use LMDB. You're likely to hit "max readers limit" if you don't fully understand how your "application" manage transactions and your "runtime" manage threads.

1. I'm assuming you're running an web application because a single threaded application wouldn't have these problems.

[davidwynter]

Hmmm, it seems I had not solved this issue. I tried the setMaxReaders, it solved the problem. According to the docs I them to 1024 for my main environment. Is there advice on how many we might need. The Kontraktor framework I am using is a single thread Actor based message passing system. It supports about 10,000 websocket calls per second per web app. Each of these is backed by a read from lmdbjava, a large environment, about 1 in 5 is a write to a small environment. Any guidance?

[cardamon]

For reference, here's a conversation on Twitter where the LMDB author explains the consequences of using mdb_env_set_maxreaders: https://twitter.com/armon/status/534867803426533376

[chriswyatt]

Interesting. Thanks. I decided to set it low so it would error more quickly if transactions weren't being closed when debugging, so I guess that might be another good reason not to set it too high? I'm keeping it low for experimental purposes right now.

[benalexau]

Did you manage to find answers to your questions in the end?

In my applications I tend to use JVM-based locking primitives and therefore minimal LMDB-side transactions. While this avoids JNR calling overhead, the main reason is I find it easier to provide Java logic that avoids (a) LMDB file growth if there is a read transaction at the same time as a write transaction and (b) the thread blocking that occurs if a second write transaction is attempted before the first write transaction completes. Certainly this relies on the luxury of not needing multi-process use, and I still depend on LMDB transactions for atomic units of work (but this is generally on a batch basis, typically a time series boundary).

[bluegol]

Just to share my 2 cents.

I think MDB_NOTLS should be used when there are no dedicated threads handling lmdb tasks. (I think there are few cases in Java apps where dedicated threads are used.) Then I use semaphores to control the number of readers, which corresponds to either Env or Txn, depending on how you use lmdb. This solved the problem for my case.

BTW, info().numReaders does NOT represent the current number of readers. (The doc is terse here. I confirmed this by reading the source.)

[jbee]

FYI: I just encountered this problem too. I did not set setMaxReaders either. After the first read and write transaction I got the error when new read should be opened. After setting it the problem disappears. I lowered the number down to 1 and it was still fine. Removed the setMaxReaders again and still working. Now I am confused.

I tested a bit more starting with a new DB (non existing file). Without the setting I get the error on 2nd/3rd TX. I increased the setting starting from 1. At 4 I get the error for a while and than it works for a while if I just hit refresh on my page that causes 2 TXs, one read and one write. But overall I can redo this endlessly so I do not miss to close a TX I guess. Before 4 it is less working more error but sometimes I get lucky. I increased to 8 and the error does not occur at all even if I hit refresh very fast. Than removing the setting again will not make the error reoccur. There must be something stored in the DB. Like the setting is never made smaller than before only larger. I confirmed this by reading the setting from the EnvInfo it stays at 8 even if I set it to 4 on next start. Also I see that the numReaders increases to e.g. 4 if I refresh my page 4 times. If I do it once per second it could stay at 4. If I do it faster it goes to 6 and stays at 6. Even if I don't do something for a long time it stays at 6. Is this expected or a problem?

[jbee]

Can anybody comment on if I am doing something wrong or if some "open" readers are "normal"?

[krisskross]

I have never had this problem. But I can try to reproduce the problem if you post the code for a self-contained program.

[jbee]

When I tried to reproduce it in a unit test I noticed that the very same code would not cause this unless I do it in a different thread than the one created the Env. Does this ring any bells?
I do openDbi once on construction, and use the Dbi instances from multiple threads. Documentation I found suggests it should be done this way.

One hint I found was that numReaders is not the actual/current number or readers but the maximum number of concurrent readers used so far. So LMDB might just use a new reader when different threads are used up to a point when it tries to reuse them?

[krisskross]

Which thread created the Env shouldn't matter. Sounds like a race condition maybe?

Sorry, I don't have any concrete advice. But my suggestion is to start from the "simplest thing that could possible work" and work your way towards your target design in small increments. Start single threaded. Preferably from a clean psvm. Try sync startup first. Go parallel only when you feel confident that all parts works as expected.

[jbee]

As I said I can run the very same code single threaded and numReaders stays at 1. The multi-threading is due to the web workers hitting that code. Most likely there is a worker pool. Maybe each worker thread gets its reader for simplicity? However, for now it does not seamed to be an issue. I will let you know if I find out more but I stop digging for now.

Can you elaborate on what sync startup means?

[benalexau]

Was this ever resolved? It's been over three months since the last comment so I'd like to close the ticket if possible.

[lfoppiano]

Dear everybody, what should be the correct approach in case you have a web application with a javascript frontend that perform asynchronous requests?

In my case, I'm ending up with half of the requests randomly failing for max reader error.
I've configured the database using NOTLS and I'm only reading, but a query web request could be composed by one or two transaction on different databases.

Any advice?

Thanks
Luca

[bluegol]

@lfoppiano,

I had similar problems.

I admit I haven't looked at lmdb source files carefully, but it seemed difficult to me to corporate with lmdb from java side. So I ended up managing reader/writer for lmdb Env myself: the reader/writer count is managed by my program using semaphore. If the count is 0, my program opens lmdb Env; otherwise it just returns already open Env. If the count drops to 0, my program closes lmdb Env.

This may not be a satisfactory answer, but this works (obviously).

[jbee]

I had similar problems.

I discovered that numReaders by default is 1 (see above comments). After initializing LMDB like this

Env.create().setMaxReaders(8)...

the problems disappeared. I assume that this means that now 8 parallel threads can hit before you get the error. Note however, that once this has been set to a number it somehow can only be set higher but never again lower. I never got or found an explanation for it but it solved my problem.

[lfoppiano]

@bluegol thanks for your answer. I had already some problems related to it on different parts of my application when I didn't close transactions correctly. At this time everything seems to be correct.

@jbee thank you for your answer. Yes I've thought about it but first I want to be sure about the cause.

@krisskross, @chriswyatt have you got any idea/hint I could investigate?

[benalexau]

The Env.Builder.setMaxReaders(int) seems to regularly cause confusion, so I think we should either make it reflect the LMDB default of 126 slots or at least discuss setting the value higher in the tutorial. @krisskross any preference on this?

[krisskross]

Oh, I didn't realize that the tutorial had maxReaders set to 1. That could indeed be a source of confusion. I'd say remove it from the tutorial or set it to 126.

Increasing the maxReaders default internally feels wrong to me.

[lfoppiano]

OK, 126 shouldn’t be the default value? I don’t set the value (as I’ve repeatedly seen it’s not good practice) but it seems to be set to 1 by default.

…

On 12 Jun 2018, at 09:21, Kristoffer Sjögren ***@***.***> wrote: Oh, I didn't realize that the tutorial had maxReaders set to 1. That could indeed be a source of confusion. I'd say remove it from the tutorial or set it to 126. Increasing the default (or event setting maxReaders) internally feels wrong to me. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#65 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAA8QqTSQXlSUSopi9o30DMG5GuhRnpZks5t72wQgaJpZM4O4vY2>.

[krisskross]

I was almost certain lmdbjava didn't set values other than requested by the user. But indeed -- if maxReaders is never set we force maxReaders to 1 for the Env.

What's your thoughts on this @benalexau? I'm leaning towards removing default values, including the other lmdbjava default values present in Env.Builder.

[benalexau]

It seems a question of ergonomics, the challenge being the Env.Builder offers a builder-style pattern that hides the individual LMDB C function calls. If we eliminate defaults from the builder, we will need to introduce tracking of which mutators were invoked by the user (eg use an Integer so we can check for null) and only invoke the subset of applicable C functions.

The past ~2 years have shown people rarely have difficulty with selecting a suitable map size or number of DBIs, but there are recurring questions around reader count and impact on memory or performance. As such I think it's worthwhile to modify the builder to a reasonable default value (126) and demonstrate setting a "high" value in the tutorial so users can be more reassured.

More generally I suggest a wiki page like "Application Architecture with LmdbJava" would be well-received and can cover many rarely-changing, high-level concepts (eg threading, multiple Envs, capacity planning, file growth, key design, buffer choices, serialization, blocking transactions, performance tips etc) better than the tutorial, the latter of which can be reserved for exactly how to use the API and benefit from the CI testing to ensure ongoing accuracy.

[krisskross]

Sounds good to me.

[benalexau]

There is now a separate ticket (#102) to correct the maxReaders default, so I will close this ticket.