Permalink
Switch branches/tags
chronicle-queue-5.16.9 chronicle-queue-5.16.8 chronicle-queue-5.16.7 chronicle-queue-5.16.6 chronicle-queue-5.16.5 chronicle-queue-5.16.4 chronicle-queue-5.16.3 chronicle-queue-5.16.2 chronicle-queue-5.16.1 chronicle-queue-5.16.0 chronicle-queue-4.16.5 chronicle-queue-4.16.4 chronicle-queue-4.16.3 chronicle-queue-4.16.2 chronicle-queue-4.16.1 chronicle-queue-4.16.0 chronicle-queue-4.15.4 chronicle-queue-4.15.3 chronicle-queue-4.15.2 chronicle-queue-4.15.1 chronicle-queue-4.15.0 chronicle-queue-4.6.110 chronicle-queue-4.6.109 chronicle-queue-4.6.108 chronicle-queue-4.6.107 chronicle-queue-4.6.106 chronicle-queue-4.6.105 chronicle-queue-4.6.104 chronicle-queue-4.6.103 chronicle-queue-4.6.102 chronicle-queue-4.6.101 chronicle-queue-4.6.100 chronicle-queue-4.6.99 chronicle-queue-4.6.98 chronicle-queue-4.6.97 chronicle-queue-4.6.96 chronicle-queue-4.6.95 chronicle-queue-4.6.94 chronicle-queue-4.6.93 chronicle-queue-4.6.92 chronicle-queue-4.6.91 chronicle-queue-4.6.90 chronicle-queue-4.6.89 chronicle-queue-4.6.88 chronicle-queue-4.6.87 chronicle-queue-4.6.86 chronicle-queue-4.6.85 chronicle-queue-4.6.84 chronicle-queue-4.6.83 chronicle-queue-4.6.82 chronicle-queue-4.6.81 chronicle-queue-4.6.80 chronicle-queue-4.6.79 chronicle-queue-4.6.78 chronicle-queue-4.6.77 chronicle-queue-4.6.76 chronicle-queue-4.6.75 chronicle-queue-4.6.74 chronicle-queue-4.6.73 chronicle-queue-4.6.72 chronicle-queue-4.6.71 chronicle-queue-4.6.70 chronicle-queue-4.6.69 chronicle-queue-4.6.68 chronicle-queue-4.6.67 chronicle-queue-4.6.66 chronicle-queue-4.6.65 chronicle-queue-4.6.64 chronicle-queue-4.6.63 chronicle-queue-4.6.62 chronicle-queue-4.6.61 chronicle-queue-4.6.60 chronicle-queue-4.6.59 chronicle-queue-4.6.58 chronicle-queue-4.6.57 chronicle-queue-4.6.56 chronicle-queue-4.6.55 chronicle-queue-4.6.54 chronicle-queue-4.6.53 chronicle-queue-4.6.52 chronicle-queue-4.6.51 chronicle-queue-4.6.50 chronicle-queue-4.6.49 chronicle-queue-4.6.48 chronicle-queue-4.6.47 chronicle-queue-4.6.46 chronicle-queue-4.6.45 chronicle-queue-4.6.44 chronicle-queue-4.6.43 chronicle-queue-4.6.42 chronicle-queue-4.6.41 chronicle-queue-4.6.40 chronicle-queue-4.6.39 chronicle-queue-4.6.38 chronicle-queue-4.6.37 chronicle-queue-4.6.36 chronicle-queue-4.6.35 chronicle-queue-4.6.34 chronicle-queue-4.6.33 chronicle-queue-4.6.32
Nothing to show
Find file Copy path
Fetching contributors…
Cannot retrieve contributors at this time
executable file 291 lines (172 sloc) 16.1 KB

Frequently Asked Questions about Chronicle

What is Single Chronicle Queue?

Single Chronicle Queue is designed to be easier to work with compared with the previous vanilla and indexed Chronicle versions.

It supports:

  • concurrent readers in different JVMs.

  • concurrent writers in different JVMs.

  • writing raw bytes or use a wire format to make schema changes, and to simplify dumping the data.

  • rolling files; weekly, daily, or even more often.

  • storing key information in the header so that there is no longer the requirement that the writers and readers be configured the same.

  • not needing to know the size of the message.

What is the performance like?

Single Chronicle (single file per cycle) supports sub-microsecond latencies. If you use Wire, the typical latencies tend to be around one micro-second. You can still use raw writing of bytes if you need maximum performance.

With a tight reader loop, I see 100% utilization. Will there be processing capability left for anything else?

Two approaches for reducing CPU usage are;

  • combine tasks into the same thread. EventGroup in Chronicle threads helps to do this dynamically.

  • use a Pauser such as a LongPauser to control how a thread backs off if there is nothing to do. There is a PauseMonitor to allow you to periodically print how busy each thread is.

What is Chronicle designed for?

Chronicle is design to record everything of interest in the logger and persisted IPC. A key requirement of low latency systems is transparency, and you can record enough information to enable a monitoring system to recreate the state of the monitored system. This allows downstream systems to record any information they need, and perform queries without having to touch the critical system.

Chronicle works best in SEDA style event-driven systems, where latency is critical, and you need a record of exactly what was performed, and when. All this can be done without expensive network sniffing/recording systems.

How does Chronicle Queue use memory?

Chronicle Queue is designed to use virtual memory which can be much larger than main memory (or the heap). This is done without a significant impact on your system, and allows you to access the data at random, quickly.

Article with an example of a process writing 1 TB in 3 hours.:

https://vanilla-java.github.io/2017/01/27/Chronicle-Queue-storing-1-TB-in-virtual-memory-on-a-128-GB-machine.html

This example shows how much slower it gets as the queue grows. Even after it is 1 TB in size, on a machine with 128 GB, it can still consistently write 1 GB in under 2 seconds.

While this does not cause a technical problem, we are aware this does concern people who can find this out-of-the-ordinary. We plan to have a mode which reduces virtual memory use; even if it is a little slower for some use cases.

What was the library originally designed for?

The original design was for a low-latency trading system which required persistence of everything in, and out, for a complete record of what happened, when, and for deterministic testing. The target round-trip time for persisting the request, processing, and persisting the response, was 1 micro-second.

Key principles are:

  • ultra-low garbage collection; less than one object per event

  • lock-less

  • cache friendly data structures.

What was not in the originally design?

The marshalling, de-marshalling, and handling of thread-safe off-heap memory, has been added more recently, as well as moving into the Java-Lang module.

This library now supports low-latency/GC-less writing, reading/parsing of text, as well as binary.

How fast is fast?

Chronicle is designed to persist messages, and replay them in micro-second time. Simple messages are as low as 0.4 micro-seconds. Complex messages might take 10 micro-seconds to write and read.

Chronicle is designed to sustain millions of inserts and updates per second. For bursts of up to 10% of your main memory, you can sustain rates of 1 - 3 GB/second written.

For example,l a laptop with 8 GB of memory might handle bursts of 800 MB at a rate of 1 GB per second. A server with 64 GB of memory might handle a burst of 6.5 GB at a rate of 3 GB per second.

If your key system is not measuring latency in micro-seconds, and throughput in thousands-per-second, then it is not that fast.

How does it scale?

It scales vertically. Many distributed systems can scale by adding more boxes. They are designed to handle between 100 and 1000 transactions per-second, per-node. Chronicle is design to handle more transaction per-node, in the order of 100K to 1M transactions per second. This means that you need far fewer nodes; typically, between 10 and 100 times fewer.

Vertical scalability is essential for low latency, as having more nodes usually increases latency.

Having one node which can handle the load of a data centre can also save money, and power consumption.

What if I have a slow consumer?

Chronicle has an advantage over other queuing systems, in that the consumer can be any amount behind the producer; up to the free space on your disk.

Chronicle has been tested where the consumer was more than the whole of main memory behind the producer. This reduced the maximum throughput by about half. Most systems, in Java, where the queue exceeds the size of main memory, cause the machine to become unusable.

Note
The Consumer can stop, restart, and continue, with minimal impact to the producer, if the data is still in main memory.

Having a faster disk sub-system, helps in extreme conditions like these. Chronicle has been tested on a laptop with an HDD with a write speed of 12 MB/s, and an over-clocked hex core i7 PCI-SSD card, which sustained a write speed of 900 MB/s.

Can Chronicle Queue be used like RMI?

It is possible to use Chronicle Queue to invoke a method on the other JVM, and wait for the return value.

However, this could be overkill, especially if you do not have to keep the history of the request and responses.

Imagine a simple scenario with two processes: C (client) and S (server). Create two `SingleChronicleQueue`s:

  • Q1 for sending requests from C to S

  • Q2 for sending responses from S to C

The server has a thread that is polling (busy spin with back-off) on Q1. When it receives a request:

  • with id=x it does whatever is needed, and writes out response to Q2

  • with id=x. C polls Q2 with some policy, and reads out responses as they appear. It uses the id to tie responses to requests.

The main task would be in devising a wire-level protocol for serialising your commands (equivalent to the method calls) from the client. This is application-specific, and can be done efficiently with the Chronicle tools.

Another issues to consider is what should the client do with the historical responses on startup? Some heartbeat systems so that the client knows that the server is alive. Archiving the old queues (VanillaChronicle) makes it easier, but at some cost.

For more details on how to do this read this series of posts

What types of Excerpt are there?

Chronicle has three types of excerpt, each optimised for different purposes.

ChronicleQueue queue = SingleChronicleQueueBuilder.binary(basePath).build();
ExcerptAppender appender = queue.acquireAppender(); // sequential writes.
ExcerptTailer tailer = queue.createTailer();       // sequential reads ideally, but random reads/write also possible.

How does writing work?

You can write using a try-with-resource block:

try (DocumentContext dc = wire.writingDocument(false)) {
    dc.wire().writeEventName("msg").text("Hello world");
}

You can write using a lambda which describes the message:

appender.writeDocument(wire -> wire.write(() -> "FirstName").text("Steve")
                                   .write(() -> "Surname").text("Jobs"));

For example. you may want to write different types of messages to a Chronicle queue, and process messages in consumers depending on their types. Chronicle Queue provides low level building blocks so that you can write any kind of message; it is up to you to choose the right data structure.

For example, you can prefix the data that you write to a Chronicle queue with a small header, and some meta-data. You can then use it as a discriminator for data processing.

You can also write/read a generic object. This will be slightly slower than using your own schema, but is it a simple way to always read the type you wrote.

How does reading work?

When you read an excerpt, it first checks that the index entry is there; the last thing that was written.

try (DocumentContext context = tailer.readingDocument()) {
    if (context.isPresent()) {
        Type t = tailer.read(() -> "message").object(Type.class);
        process(t);
    }
}

How is disk space managed?

A key assumption is that disk space is cheap, or at least it should be. Some organizations have unrealistic internal charging rates, but you should be able to get 100 GB for about one hour of your time. This assumes retail costs for disks compared with minimum wage. The organizational cost of disk is often 10-100x the real cost, but so is your cost to the business.

In essence, disk space should be cheap, and you can record between a week and a month of continuous data, on one cheap drive.

Never the less, there is less maintenance overhead if the Chronicle logs rotate themselves, and there is work being done to implement this for Chronicle 2.1. Initially, Chronicle files will be rotated when they reach a specific number of entries.

I want to use Chronicle as an off-heap cache. What do I do?

Chronicle Queue is designed for replay. While it can be used as an off-heap persisted cache, it does not do this very easily. Chronicle Map is likely to be a better choice for this requirement.

Can records be updated?

They can be updated at any time, but you lose any event-driven notification to readers at this point. It might be practical to have multiple Chronicle queues; one which stores large updated records, and another for small notifications.

I want to store large messages; what is the limit?

The limit is about 1 GB, as of Chronicle 4.x. The practical limit without tuning the configuration is about 16 MB. At this point you get significant inefficiencies, unless you increase the data allocation chunk size.

I get an exception writing an excerpt. What does this mean?

The message will be lost, and it is truncated.

I get an exception attempting to read an excerpt. What does this mean?

Most likely your read code does not match your write code. Using Wire means it can handle changes to fields, and to data types, transparently.

How does the byte order work with replication?

The byte order does not change with replication. This means that it will work best in byte-endian homogeneous systems such as Windows/Linux x86/x64/ARM. Chronicle may support changing the byte order in future.

Does Chronicle support other serialization libraries?

Chronicle Queue supports CharSequence, Appendable, OutputStream and InputStream APIs. It also has a fast copy to/from a byte[] and ByteBuffer.

Chronicle Queue is designed to be faster 'with persistence' than other serialization libraries are even 'without persistence'. Chronicle Queue supports YAML, JSON, Binary YAML, and CSV. To date, we have not found a faster library for serialization without a standardized format. Chronicle does not yet support XML.

Where XML is needed downstream, we suggest writing in binary format, and have the reader incur the overhead of the conversion, rather than slow down the producer.

Does Chronicle support a synchronous mode?

Chronicle Queue v4.x does not currently support a synchronous mode. The best approach is to wait for a replicated message to be acknowledged.

Can Chronicle Queue compete with Spark in this domain?

To our knowledge, Spark Streaming is designed for real-time, but is looking to support a much lower message rate, and does not attempt to be ultra-low GC (minor GC less than once a day). We have not heard of any one using Spark in the core of a trading system. It tends to be used for downstream monitoring and reporting.

It seems that you have some clients that use Chronicle Queue for Big Data style problems.

Our largest Chronicle Queue client pulls in up to 100 TB into a single JVM using an earlier version of Chronicle Queue.

Could you please us more about the way they use Chronicle Queue?

Chronicle Queue is compelling as it uses a no-flow control model. Chronicle Queue is designed to not slow down the producer if you have a slow consumer. Instead, you need to give it plenty of disk space as a buffer. Disk space is cheaper than main memory, and is cheaper than heap space. You can buy a system with multiple 16 TB SSD drives today. No one would consider having a JVM heap with 100 TB.

A couple of prime examples are

  • Market data consumers. You cannot use flow control with an exchange.

  • Compliance. It Is something you have to have, but systems which send data to compliance, never want to be slowed down by it.

In Chronicle Queue v4, will an error such as IllegalStateException appear when there is a high number of messages to write?

Chronicle Queue v4+ does not have the limitation of using just one thread. It supports any number of threads, with a single file per cycle.

The recommended pattern for implementing a listener pattern, is to use the methodReader/methodWriter which can also take care of timestamps when.

For example, you may want a built-in Chronicle queue mechanism for asynchronous 'appender → tailer' notifications, where, upon receipt of a notification event, a given tailer is guaranteed to have at least one entry posted by appender ready for read. For the tailer, the only way it knows there is a message, is by reading/polling the end of the queue. If the appender and tailer are in the same process, you can use a different mechanism of your choice.

We would suggest you read these https://vanilla-java.github.io/tag/Microservices/ from the bottom up starting with part 1.

What would cause Chronicle Queue to segfault?

It could be a race condition. When a memory mapping is truly freed, it cannot be accessed, or it will trigger a segmentation fault. The reason to suspect this, is that it should be free on a roll from one cycle to the next.

Thread safety

Can I have multiple readers?

A given Chronicle queue can safely have many readers, both inside and outside of the process creating it.

To have multiple readers of a Chronicle queue, you should generally create a new Chronicle queue per-reader, pointing at the same underlying journal. On each of these Chronicle queues, you will call createTailer and get a new tailer that can be used to read it. These tailers should never be shared. A less performant option, is to share a single Chronicle queue and tailer, and lock access with synchronized or ReentrantLock. Only one tailer should ever be active at the same time.

Can I have multiple writers?

You can have any number of writers. However, you may get higher throughput if you have only one writer at a time. Having multiple writers increases contention, but works as you would expect.

Replication

Does Chronicle Queue support replication?

Replication features are availabler in Chronicle Queue Enterprise. Replication supports:

  • replication of a single master to multiple slave nodes.

  • writers can wait for replication to be acknowledged.

  • readers can wait to only read acknowledged messages.

  • throttling and traffic shaping.

Does Chronicle Queue support UDP replication?

No. Chronicle Queue is designed to be both reliable and deterministic. UDP is not designed for this. A hybrid UDP/TCP system is possible in the future.

How do I know the consumer is up to date?

For the tailer, either replicated or notreplicated, you can assume you are up-to-date when either isPresent() is false, or your read method returns false