Insert time keeps going up

[xieu90]

Hi everyone,
I tried to insert a lot of data into eclipsestore and observed the time.

public class MyEclipseStoreRoot {
    private List<Adress> adressRoot = new ArrayList<>();
    private List<Book> bookRoot = new ArrayList<>();
    private List<Author> authorRoot = new ArrayList<>();
    private List<Relation> relationRoot = new ArrayList<>();
}


public class Book {
    private String name;
    private LocalDate publishDate;
}

each time I will try to insert 500 000 objects (in this case only books), the storemanager.store will be called on each 1000 (chunk)
as I observed the first time (500k) each chunk will take about 6ms to store, then it will go up to about 100ms at the end. for whole 500k it would take 25595 milliseconds for first time = about 25s

second time: immediately 100ms for each chunk, later: 200ms total:74958 milliseconds for second 500k = 80s
third time: immediately 300ms for each chunk, immediately saw 500(rarely), standard: around 200-300, later: around 300 total 144822 milliseconds = 145 s
fourth time: immediately 390ms for each chunk, stay around: 380-400ms sometimes 500 total 219017 milliseconds = 3,65 min
fifth time: i messed up a bit and had to restart computer
fifth time: immediately start at 600 climb up to 700 for each 1000/chunk total 305916 milliseconds 5 min

just used the default normal storemanager with root and path, nothing special (didnt set channel)

so i ended up at about almost 3 million object in eclipsestore. the insert time keep going up the more data one have. if in future i have some billion objects then some 10 000 min to insert 1 new million data.
are there any way to keep the time like the first time (1 digit ms for 1000 objects in chunk)
in database (sql family) i heard that indexing might help. In Microstream/Eclipsestore I have no idea.

[ppazos]

From what I read, each time you save. the graph gets versioned, maybe the extra time has to do with that, if you have all the books inside the same root object, maybe the versioning is affecting the time. I think another thing that might be affecting that is the available memory, again from what I read, it seems the store is done in memory and then persisted to disk, the memory part is fast but the IO for the disk might affect execution time.

Maybe if you publish your full code, someone can spot something that might guide you in the right direction.

[guykatz]

i am interested in ths answer...
@xieu90 can you post the code of how you store?

[xieu90]

hi, sorry for late reply ^^
the full source code was uploaded to https://github.com/xieu90/eclipsestore/tree/main

where crud and batching live:
https://github.com/xieu90/eclipsestore/blob/main/src/main/java/de/own/eclipsestore/service/MyService.java
data root
https://github.com/xieu90/eclipsestore/blob/main/src/main/java/de/own/eclipsestore/entity/MyEclipseStoreRoot.java
Books, Author, Relations are in entity package
MainView is just a ui with buttons, so that i can click and call the MyService to spam actions (mostly count,generate authors, books and relation)
https://github.com/xieu90/eclipsestore/blob/main/src/main/java/de/own/eclipsestore/controller/MainView.java

this one and MyService are where and how i stored things

ah, i remember the size of free space on my linux system flied up too. I was shocked at first caused it took some GB, but later it will go down (may be some cleaning, household thing from eclipsestore or garrabe collector did sth) the note for that is unfortunately lost...

[useryq8]

I encounter the same issue, the microstream(the same with eclipse-store) write performance is not as good as DB, you can see my microstream-one/bookstore-demo#10, when writing the one book, the write performance is very slow comparing with DB

it seems that EclipseStore is suitable for reading, not for writing. if your application is 99% read, %1 write , it will be good choice.

[guykatz]

thats a big statement...
@fh-ms can you or someone from the team comment?

[fh-ms]

When writing frequently the same type of object, there are two ways to speed up things significantly.

1. Segmenting the data. Instead of a single List<T>, use a Map<K,List<T>> were K is the segment key, e.g. a date or batch number. So for each batch only a single list has to be written containing the new elements. An example on how to segment data can be found here.

2. Using our LazyList, where the segmentation happens internally.

[fh-ms]

Regarding the mentioned lost data after restart:

The reason is that the root is overwritten on every restart.

Correct way instead of calling setRoot every time is to initialize the EmbeddedStorageManager with a root reference.

final MyEclipseStoreRoot root = new MyEclipseStoreRoot();
EmbeddedStorageManager storageManager = EmbeddedStorageConfiguration.Builder()
        .setChannelCount(4)
        .createEmbeddedStorageFoundation()
        .createEmbeddedStorageManager(root);
storageManager.start();

Or check if no root object is present:

EmbeddedStorageManager storageManager = EmbeddedStorageConfiguration.Builder()
        .setChannelCount(4)
        .createEmbeddedStorageFoundation()
        .createEmbeddedStorageManager();
storageManager.start();

if(storageManager.root() == null)
{
	storageManager.setRoot(new MyEclipseStoreRoot());
	storageManager.storeRoot();
}

[xieu90]

Just changed MyEclipseStoreRoot to use LazyArrayList (Map and Segment will be my snack in future ^^)
and used .createEmbeddedStorageManager(root);

the result was impressive , now it just need two digit whole time
2,4 M Author , 3,5 M Book , 1M Relation

1000 authorss in chunk should be saved by eclipsestoreTime taken for only saving: 13 milliseconds (highest time, usually 1 digit)
800000 authors should be generatedTime taken: 8790 milliseconds

1000 books in chunk should be saved by eclipsestoreTime taken for only saving: 6 milliseconds
1000000 books should be generatedTime taken: 4979 milliseconds

1000 Relations in chunk should be saved by eclipsestoreTime taken for only saving: 3 milliseconds
100000 relations should be generatedTime taken: 42527 milliseconds ----- this one took some how longer than other 2, but maybe it was because of Math.random or some disk problem , but not like i care, cause i am happy with speed boost from Lazy

PS: tried it again with relation (just some button clicks anyway)
1000 Relations in chunk should be saved by eclipsestoreTime taken for only saving: 4 milliseconds
100000 relations should be generatedTime taken: 634 milliseconds

or
1000 Relations in chunk should be saved by eclipsestoreTime taken for only saving: 10 milliseconds
100000 relations should be generatedTime taken: 974 milliseconds

may be garbage collector or cleaning was in background
I guess the thread can be closed now, I have nothing more from my side. Thank you everyone ^^

[useryq8]

"1000 books in chunk should be saved by eclipsestoreTime taken for only saving: 6 milliseconds,"
hello, @xieu90, do you save 1000 books in one store call? or call 1000 times store, then calc the avg cost? in my application, every customer will call one store for his order, so the one book store cost is my focus because save 1000 customer's order in one store call can't be accepted. when otherthing happen accidentally, the customer's order data can't be lost.

[xieu90]

hi @useryq8
do you save 1000 books in one store call? yes (for Book, Author, Relation)
or call 1000 times store, then calc the avg cost? no for Book, yes for Adress

there are two types of saving - crud/single or multiple/batch.
my code originally used single save like you described, but my focus was to see performance when i already have millions items, so changed them to multiple batch to save time
If you take a look at MainView search for "generate" you will find things like Adress, Book, Relation,Author. Adress still have single save, other 3 are batch.
so the answer to that question is mostly no because i mostly asked about book, relation and author here. if you try the code for Adress then its yes ^^

I rerun the code for address the result for 100K is:
100000 adress should be generatedTime taken: 218439 milliseconds
about 3min for 100K

when saving only one:
1 adress should be generatedTime taken: 144 milliseconds
or later I usually get:
1 adress should be generatedTime taken: 1 milliseconds

usually saving one item is fast, it just depends on the load of server, how big/complex the item you want to save is and the indexing of database (in eclipsestore just use Lazy and its solved ^^)
imho in your case you dont really need to worry about performance yet. 1000 users input data take a while and they dont save all at same time.