Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Replaced OrientDB with ArcadeDB (latest version) #745

Open
wants to merge 3 commits into
base: master
Choose a base branch
from

Conversation

lvca
Copy link

@lvca lvca commented Jan 11, 2022

OrientDB is not supported anymore and has issues with Apple Silicon (see #709). ArcadeDB is a (conceptual) fork, with a new engine, much faster and lighter and no use of JNA/JNI, just 100% pure java and embeddable. Same license (Open Source - Apache 2) and with an active community.

The SQL layer is the same between ArcadeDB and OrientDB, so no need to rewrite the queries.

Run the test cases and it's all green.

@mhberger
Copy link

Cool! That is a lot less invasive.

Have tried kicking the wheels. My primary use case is to install the
distribution binary (either download or using something like sdkman).
Then test/check with simple blog (which has a custom type) and jbake.org.

In this case we run the build to generate the binaries and use those.

Doing the build

../gradlew clean test smokeTest build

Has one failing test.

  OvenTest > shouldBakeWithRelativeCustomPaths() FAILED

However, we want the zip file so the following does it.

../gradlew clean assemble

Installing and running to generate simple blog on iMac Pro Intel produces
“Baked 47 items in 3050ms” and on MacBook Pro M1 Max “Baked 47 items in 1777ms”.
Note to run from a known state, I have to

rm -rf cache

I wonder if there is an option to not use the cache or clean it out the
start of a run?

Installing and running to generate jbake.org generates the following error on both iMac
Pro Intel and MacbBook Pro M1 Max.

4:52:30.631 ERROR jbake - An unexpected error occurred: Record too big to be stored in bucket 'Documents_4' (5), size=65008 max=57342. Change the `arcadedb.bucketDefaultPageSize` database settings and try again

Where is a good place to change this setting? I tried jbake.config.

@lvca
Copy link
Author

lvca commented Jan 11, 2022

@mhberger I see one difference is that some tests used "memory:" storage type of OrientDB, but in Arcade that's not available: only persistent databases with ArcadeDB. Could it be the database is not deleted between tests from the file system?

I've seen the same test failing, but if I run it separately passes.

You can set this at startup (before the database is created/opened): GlobalConfiguration.BUCKET_DEFAULT_PAGE_SIZE.setValue(131072) or by defining a setting or env variable java ... -Darcadedb.bucketDefaultPageSize=131072

NOTE: The page size is a temporary limitation. In the future, if a record is larger than a page, it will be managed transparently without the need to set the page size.

@mhberger
Copy link

mhberger commented Jan 11, 2022

Thanks! Running with the following does the trick. And for the note that
it should not be required in the future.

rm -rf cache
export JAVA_OPTS=' -Darcadedb.bucketDefaultPageSize=131072'
jbake -b

Some rough timings.

Site        | Hardware            | Database   | Bake Info
----------- | ------------------- | ---------- | ----------
simple blog | iMac Pro Intel      | OrientDB   | Baked 47 items in 4044ms
simple blog | iMac Pro Intel      | SQLite     | Baked 47 items in 2839ms
simple blog | MacBook Pro M1 Max  | SQLite     | Baked 47 items in 1836ms
simple blog | MacBook Pro M1 Max  | ArcadiaDB  | Baked 47 items in 3019ms
            |                     |            |
jbake.org   | iMac Pro Intel      | OrientDB   | Baked 196 items in 10731ms
jbake.org   | iMac Pro Intel      | SQLIte     | Baked 196 items in 15461ms
jbake.org   | iMac Pro Intel      | ArcadiaDB  | Baked 196 items in 12882m
jbake.org   | MacBook Pro M1 Max  | SQLite     | Baked 196 items in 9611ms
jbake.org   | MacBook Pro M1 Max  | ArcadiaDB  | Baked 196 items in 7636ms

@lvca
Copy link
Author

lvca commented Jan 11, 2022

Not bad as the first implementation. We can do much better. By the way, I've set this in the constructor, so no need to set anything anymore:

// USE A 4X BIGGER PAGE THAN THE DEFAULT
GlobalConfiguration.BUCKET_DEFAULT_PAGE_SIZE.setValue(Bucket.DEF_PAGE_SIZE * 4);

Also, I see Document records have a type property. What types are supported? My idea is to use the ArcadeDB polymorphism and create a subtype of Document, like:

Post <--- Document

So you can avoid indexing the type and the queries would be faster. Instead of doing select * from Documents where type='%s' order by date desc you could do select * from %s where order by date desc.

Unless type is a dynamic property that can change in a Document or unless you can have arbitrary types created at run-time, using the polymorphism helps. And of course, if you do select from Document, all the records of Document and sub-types will be returned.

@lvca
Copy link
Author

lvca commented Jan 31, 2022

Hi guys, any plans to replace OrientDB with ArcadeDB?

@lprimak
Copy link

lprimak commented Feb 22, 2022

It's still failing one test :( I would love to run this on m new CI server (which is broken at the moment, since it's M1 :)

@lvca
Copy link
Author

lvca commented Feb 22, 2022

@lprimak what test is failing? I've got a MacBook Pro 16 Intel and everything passes:

BUILD SUCCESSFUL in 1m 1s
18 actionable tasks: 3 executed, 15 up-to-date

@lprimak
Copy link

lprimak commented Feb 22, 2022

Well it fails Travis CI that's set up for the project here.

@lvca
Copy link
Author

lvca commented Feb 22, 2022

I see, the failing test is:

@Test
public void shouldBakeWithRelativeCustomPaths() throws Exception {
  sourceFolder = TestUtils.getTestResourcesAsSourceFolder("/fixture-custom-relative");
  configuration = (DefaultJBakeConfiguration) new ConfigUtil().loadConfig(sourceFolder);
  File assetFolder = new File(configuration.getDestinationFolder(), "css");
  File aboutFile = new File(configuration.getDestinationFolder(), "about.html");
  File blogSubFolder = new File(configuration.getDestinationFolder(), "blog");


  final Oven oven = new Oven(configuration);
  oven.bake();

  assertThat(oven.getErrors()).isEmpty();
  assertThat(configuration.getDestinationFolder()).isNotEmptyDirectory();
  assertThat(assetFolder).isNotEmptyDirectory();
  assertThat(aboutFile).isFile();
  assertThat(aboutFile).isNotEmpty();
  assertThat(blogSubFolder).isNotEmptyDirectory();
}

It looks like this is false assertThat(aboutFile).isFile();. How this can be related to the underlying ArcadeDB?

@jonbullock
Copy link
Member

Apologies for the delay in getting back to you on this, I need to think about how best to manage the transition from OrientDB to ArcadeDB for users before I merge this in.

@lvca
Copy link
Author

lvca commented Mar 10, 2022

By the way, we have an importer if needed: https://docs.arcadedb.com/#OrientDB-Importer. You can also call it via API:

File databaseDirectory = new File(DATABASE_PATH);
URL inputFile = OrientDBImporterIT.class.getClassLoader().getResource("orientdb-export-small.gz");
OrientDBImporter importer = new OrientDBImporter(("-i " + inputFile.getFile() + " -d " + DATABASE_PATH + " -s -o").split(" "));
importer.run().close();

@yeroc
Copy link

yeroc commented Jun 2, 2022

@jonbullock is this PR the best way forward for Apple Silicon (M1) users?

@jonbullock
Copy link
Member

Thanks for the pointer about the import @lvca I haven't forgotten about this, I do want to explore this further but my time is extremely limited at present.

@yeroc PR #765 should buy the required time for me to explore this further.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants