Question - Is it safe to remove(...) entries while iterating over an OakMap

[dynaxis]

Java's collections all don't allow modifying one while iterating it, resulting in a ConcurrentModificationException. If I want to remove some entries from an OakMap while iterating through them, what is the proper way of achieving it?

For instance, I created a head map, tried to call clear() on it, and failed with UnsupportedOperationException. Iterator.remove is not implemented for those returned from an OakMap. And glancing the source code, I'm not quite sure if I can safely invoke remove on an OakMap, while iterating over its content.

Currently I'm iterating over the entries and calling remove immediately after I'm done with each entry. Is it ok to do so?

[sanastas]

Hi @dynaxis ,

Sorry for late reply, I was out of office.
Thanks for being interesting in Oak, here are my answers:

Generally it is safe to remove an entry while iterating over the map. However, you will not be able to access that entry via the buffer you have, as it is deleted. If you are using OakUnscopedBuffer to access that entry, the access that happens after deletion will cause exception (also on other threads accessing same memory). If you are using OakUnsafeDirectBuffer, you will be able to access, but this is unsafe and may lead to errors, as memory can be reallocated for other use.

But as you say, you delete entry after finishing dealing with it, so it should be fine.
As you noticed, the API Iterator.remove() is not yet implemented in Oak. Not that it can not be done, we just didn't get to it yet. If you would like to contribute to our open source (to add this API), we can guide you with all needed information :)

Will be happy to answer any other questions!
Thanks,
Anastasia

[dynaxis]

@sanastas Thank you for the answer. I'm currently using Oak for one of my constracted projects, where I want to avoid pulling in something like Redis. So far, it seems doing its job great. Thank you for the great software, and as you stated, I'll try to contribute back as my understanding on Oak internals.

[sanastas]

@dynaxis , sounds great!

If you can explain a bit more your use-case, it will help us to understand our users more. Not urgent, when you have time.
And if you want to know more about Oak internals, let me know. Thanks!

[dynaxis]

I use Oak in a web log processor, which takes from and writes to Kafka topics and eventually to a database. In the course, it tracks the last visited page for each session to decorate logs w/ their previous visits and other contextual information.

Since I wanted it to cope with failures, it creates an OakMap per each Kafka topic partition. And on partition reallocations, OakMaps are created and their states are restored by replaying some of previous logs only for the reallocated partitions (I use log compaction to reduce the amount of logs to replay. Using the topic to mimic WAL with snapshots). As you see, it's single-threaded application in terms of accesses to an OakMap.

There is one more place levaraging OakMap. Old sessions should be evicted to free up memory, and I use one OakMap for each interval of an hour from the current time to, say, 24 hours ago, and organizes sessions into them according to the last log for the sessions. So every hour, I can delete sessions in the OakMap holding the sessions inactive for more than 23 hours.

I didn't want to pull in Redis for this since it is natural and considered efficient to have an in memory map in process. So I picked Oak and am happy with the decision.

[sanastas]

@dynaxis thank you so much for this detailed explanation!