JsonDB storage service #1868

cdjackson · 2016-07-15T19:55:31Z

This is an implementation of a json storage service for initial comment. Its main advantage is that it allows a user to edit the files with a text editor or json editor and is easily human readable. It is also considerably faster than mapdb. It has the potential to be less robust, but given it can be edited, and backups easily maintained, this is considered to be a low risk (currently!).

This implements the StorageService as a Map, but persists the data to disk as json files. Each storage that is opened is persisted to a separate file. Data is only read from the disk on startup.

The system employs a delayed write mechanism (currently 500ms) to improve performance where there are multiple writes in a short time. When the service is closed, all open services are written to disk so there should be no risk of loosing data unless the JVM crashes and the delay could be increased a little (probably still best to keep it low to reduce the chance of lost data due to a crashed JVM).

The service keeps backups of each table - currently up to 5 backups are maintained. Each time the json file is written, the old file is moved into a backup folder, and old files are removed from the backup folder.

Benchmark tests indicate that this is significantly faster than the mapdb storage service (20 to 30 times faster). With the pretty print option enabled the storage size is similar to the mapdb, and about 30% smaller if this is disabled. Currently I've left this enabled since the major advantage of this service is that the data can be edited easily so improving its readability at the expense of some disk space is considered acceptable.

Benchmarks

Test	MapDB	JsonDB
2000 writes	8.2 seconds	0.2 seconds
20000 writes	80 seconds	0.4 seconds

maggu2810 · 2016-07-16T07:50:06Z

The benchmark sounds interesting.

@pwolfart did IMHO a great job testing database corruption of the mapdb. If you are interested, have a look at

If your implementation is called to write very 499 ms it seems that the data is never written.
A very theoretical use case, but perhaps it should not be deferred all the time?

It is a nice feature that you can manually edit the storage on corruption, but I assume this is a feature for us / developers only. This is nothing that a normal user wants to do.
From my point of view the most important point is that we never need to fix a corrupt database because it does not corrupt at all or uses some backup strategy itself.

But IMHO we can merge such a storage backend (have not had a look at the code) and leave it to the product which one is chosen to use.

hakan42 · 2016-07-16T09:57:56Z

Honestly, I would love to have such an (optional) storage system, especially during development but also while setting up a fresh system. I just love to see what makes a system work 😄

hfanieng · 2016-07-16T10:39:05Z

For me as very experienced power-user this feature is welcome. Being beta-tester from the first version of OH2 it was a nightmare to reconfigure over 100 of items after the database-corruption.

cdjackson · 2016-07-16T10:43:39Z

If your implementation is called to write very 499 ms it seems that the data is never written.

Interesting corner case - I'll take a look. I would guess in reality this nearly never happens, but it would be good to cover.
However - the data will still be ok - it will be written when the service is deactivated, so assuming there isn't a JVM crash, it will be fine in this case at the moment.

but I assume this is a feature for us / developers only. This is nothing that a normal user wants to do.

No - I disagree. There have been a lot of discussions on the OH2 forum where people want to be able to edit the database while still having the ability to use the UI to configure the system.

cdjackson · 2016-07-16T11:33:40Z

I added a maximum deferred timer of 60 seconds, so the data will be written every 60 seconds if there was a pathological instance of a continuous write.

watou · 2016-07-16T12:14:48Z

...pse.smarthome.storage.json/src/main/java/org/eclipse/smarthome/storage/json/JsonStorage.java

+                outputStream.write(s.getBytes());
+                outputStream.close();
+            } catch (Exception e) {
+                e.printStackTrace();


smar000 · 2016-07-16T17:27:13Z

No - I disagree. There have been a lot of discussions on the OH2 forum where people want to be able to edit the database while still having the ability to use the UI to configure the system.

I fully agree with Chris on this. I think it would be invaluable to a lot of power users and not just developers.

maggu2810 · 2016-07-16T18:04:31Z

Okay, perhaps it has been misspelled using "developers" and "users".
I assume the most of the ones that added comments here are using openHAB. So an open source product that has been set up by yourself.
The Eclipse SmartHome framework is also used by commercial products.
The users I talked about are the ones that used a commercial system.

I don't think such an user is willing to edit files on his system.
The framework should ensure, that data does not get corrupt.
We cannot provide a framework that intention is also commercial usage and rely on "fixable by editing storage files".

As stated above, I think it is a nice feature and could be merged to the tree (IMHO).
Then a product can choose to use it, or use another one.

cdjackson · 2016-07-16T18:32:13Z

If the need is to have something that has higher integrity, then it would be relatively simple to add extra checks to improve the integrity by reading back from disk, and automatically reverting to the previous backup.

thesebastianf · 2016-07-31T14:26:31Z

When will this be added to OpenHAB2 :-)
many people will be happy :)

maggu2810 · 2016-09-13T14:53:29Z

bundles/storage/org.eclipse.smarthome.storage.json/about_files/LICENSE-2.0.txt

@@ -0,0 +1,202 @@
+


Which component of this PR is licensed using the APL2?

Not sure how this ended up in the repo - I’ll remove it before this is merged.

kaikreuzer

Hi @cdjackson, finally I took the time to review this! Looks good and indeed seems to work pretty well.

Just had some minor review comments, would be great if you could adapt the PR accordingly.

Please also don't miss to add this bundle to https://github.com/eclipse/smarthome/blob/master/bundles/storage/pom.xml, https://github.com/eclipse/smarthome/blob/master/features/org.eclipse.smarthome.feature.runtime.core/feature.xml#L176 and https://github.com/eclipse/smarthome/blob/master/features/karaf/src/main/feature/feature.xml#L297. Thanks!

kaikreuzer · 2016-11-07T12:09:18Z

....json.test/src/main/groovy/org/eclipse/smarthome/storage/mapdb/StorageServiceOSGiTest.groovy

+ * @author Thomas Eichstaedt-Engelen - Initial contribution
+ * @author Alex Tugarev - Added test for getStorage without classloader
+ */
+class StorageServiceOSGiTest extends OSGiTest {


I assume you changed this class (from mapdb) only for testing and accidentially committed this file?

kaikreuzer · 2016-11-07T12:49:01Z

...on.test/src/main/groovy/org/eclipse/smarthome/storage/json/JsonStorageServiceOSGiTest.groovy

@@ -0,0 +1,136 @@
+/**


could you please update the license headers to the current version?

kaikreuzer · 2016-11-07T12:50:53Z

....json.test/src/main/java/org/eclipse/smarthome/storage/json/JsonStorageServiceBenchmark.java

@@ -0,0 +1,129 @@
+package org.eclipse.smarthome.storage.json;


please add a license header

kaikreuzer · 2016-11-07T12:58:26Z

...pse.smarthome.storage.json/src/main/java/org/eclipse/smarthome/storage/json/JsonStorage.java

+
+    private final Logger logger = LoggerFactory.getLogger(JsonStorage.class);
+
+    private final int MAX_FILES = 5;


Could you make MAX_FILES, WRITE_DELAY and MAX_DEFERRED_PERIOD configuration properties of JsonStorageService, so that they can be set through ConfigAdmin?

kaikreuzer · 2016-11-07T13:02:14Z

...pse.smarthome.storage.json/src/main/java/org/eclipse/smarthome/storage/json/JsonStorage.java

+    private File getBackupFile(int age) {
+        // Delete old backups
+        List<Long> fileTimes = new ArrayList<Long>();
+        File folder = new File(file.getParent() + File.separator + "backup");


Better define "backup" as a constant.

kaikreuzer · 2016-11-07T13:09:45Z

...pse.smarthome.storage.json/src/main/java/org/eclipse/smarthome/storage/json/JsonStorage.java

+            try {
+                outputStream = new FileOutputStream(file, false);
+                outputStream.write(s.getBytes());
+                outputStream.close();


either move this to finally or use try-with-resources

kaikreuzer · 2016-11-07T13:11:38Z

...pse.smarthome.storage.json/src/main/java/org/eclipse/smarthome/storage/json/JsonStorage.java

+            commitDatabase();
+        }
+        if (deferredSince == 0) {
+            deferredSince = System.currentTimeMillis();


Better use System.nanoTime()

kaikreuzer · 2016-11-07T13:11:43Z

...pse.smarthome.storage.json/src/main/java/org/eclipse/smarthome/storage/json/JsonStorage.java

+    public synchronized void deferredCommit() {
+        // Handle a maximum time for deferring the commit.
+        // This stops a pathological loop preventing saving
+        if (deferredSince != 0 && deferredSince < System.currentTimeMillis() - MAX_DEFERRED_PERIOD) {


Better use System.nanoTime()

kaikreuzer · 2016-11-07T13:16:38Z

...pdb.test/src/main/java/org/eclipse/smarthome/storage/mapdb/MapdbStorageServiceBenchmark.java

@@ -0,0 +1,100 @@
+package org.eclipse.smarthome.storage.mapdb;


please add license header

kaikreuzer · 2016-11-07T13:16:59Z

extensions/persistence/org.eclipse.smarthome.persistence.h2sql/.project

@@ -0,0 +1,33 @@
+<?xml version="1.0" encoding="UTF-8"?>


looks like an accidential commit?

cdjackson · 2016-11-08T17:51:53Z

@kaikreuzer Hopefully this is updated to reflect all your comments.

kaikreuzer

Found a few further review comments.

Once you are done, could you please squash your commits and add an
Also-By: IVAN GEORGIEV ILIEV ivan.iliev@musala.com
line to the commit message (as two classes are copied from him?)
I'll then have to create a CQ since we exceed 1000 loc.

kaikreuzer · 2016-11-09T08:02:54Z

...rthome.storage.json/src/main/java/org/eclipse/smarthome/storage/json/JsonStorageService.java

+ *
+ * @author Chris Jackson - Initial Contribution
+ */
+public class JsonStorageService implements StorageService, ManagedService {


please do not use ManageService - use DS component configuration through activate() instead.

kaikreuzer · 2016-11-09T08:05:00Z

...rthome.storage.json/src/main/java/org/eclipse/smarthome/storage/json/JsonStorageService.java

+        }
+
+        if (properties.get(CFG_MAX_BACKUP_FILES) != null) {
+            maxBackupFiles = Integer.parseInt((String) properties.get(CFG_MAX_BACKUP_FILES));


Integer.parseInt can throw RuntimeExceptions, which you need to handle.

kaikreuzer · 2016-11-09T08:10:17Z

...pse.smarthome.storage.json/src/main/java/org/eclipse/smarthome/storage/json/JsonStorage.java

+     * Json bundle) is used in order to load the classes in the context of
+     * the calling bundle.
+     *
+     * @param json


either add some text or remove those lines

kaikreuzer · 2016-11-09T08:12:14Z

bundles/storage/org.eclipse.smarthome.storage.json/OSGI-INF/jsonstorage.xml

+	<implementation class="org.eclipse.smarthome.storage.json.JsonStorageService" />
+
+    <property name="service.pid" type="String" value="org.eclipse.smarthome.storage.json" />
+	<property name="service.config.description.uri" type="String" value="system:json_storage" />


I cannot find a config description for system:json_storage.
Anyhow, I don't think those settings need to be user adjustable through the UI - so I'd suggest to remove those service.config properties here.

cdjackson · 2016-11-09T14:48:18Z

@kaikreuzer regarding the config definition - I missed committing the file. However, I do think it's useful for people to be able to change these settings. For example, people using SSD/USB for storage may want to increase the delays to reduce the write cycles and avoid corruption. Sure, they could do it with config files, but doesn't it make more sense to have it configurable through the UI if config is available?

kaikreuzer · 2016-11-09T15:01:28Z

ok for me.

kaikreuzer · 2016-11-09T15:02:03Z

bundles/storage/org.eclipse.smarthome.storage.json/ESH-INF/config/config.xml

+
+    <config-description uri="system:json_storage">
+        <parameter name="backup_files" type="integer" min="0" max="100">
+            <label>Backup Files</label>


change to "files" to be in sync with the other labels?

cdjackson · 2016-11-10T14:12:31Z

@kaikreuzer this should be updated now I hope

kaikreuzer

Sorry, still not done...

kaikreuzer · 2016-11-10T15:28:17Z

bundles/storage/org.eclipse.smarthome.storage.json/OSGI-INF/jsonstorage.xml

+	immediate="true" name="org.eclipse.smarthome.storage.json">
+	<implementation class="org.eclipse.smarthome.storage.json.JsonStorageService" />
+
+    <property name="service.pid" type="String" value="org.eclipse.smarthome.storage.json" />


could you replace the spaces by tabs?

kaikreuzer · 2016-11-10T15:30:38Z

...rthome.storage.json/src/main/java/org/eclipse/smarthome/storage/json/JsonStorageService.java

+    protected void deactivate(ComponentContext cContext, int reason) {
+    }
+
+    public synchronized void modified(Map<String, Object> properties) {


You didn't specify the existence of the modified method in your component xml. And you shouldn't as your service is not able to deal with changes after it has been activated. Hence you should simply move this code in here to activate() and remove the modified() method.

Good point - interestingly, modified does get called ;).

kaikreuzer · 2016-11-10T15:31:05Z

...rthome.storage.json/src/main/java/org/eclipse/smarthome/storage/json/JsonStorageService.java

+
+    private final Map<String, JsonStorage<Object>> storageList = new HashMap<String, JsonStorage<Object>>();
+
+    public void activate() {


You have two activate methods now. You should move the code from here to the newly introduced one.

kaikreuzer · 2016-11-10T15:31:13Z

...rthome.storage.json/src/main/java/org/eclipse/smarthome/storage/json/JsonStorageService.java

+    }
+
+    public void deactivate() {
+        // Since we're using a delayed commit, we need to write out any data


You have two deactivate methods now. You should move the code from here to the newly introduced one.

kaikreuzer · 2016-11-10T15:31:34Z

features/karaf/src/main/feature/feature.xml

@@ -1,13 +1,4 @@
 <?xml version="1.0" encoding="UTF-8"?>


This file seems to be completely messed up - could you please check?

cdjackson · 2016-11-10T17:11:58Z

Updated again...

kaikreuzer

Almost there, I think!

kaikreuzer · 2016-11-11T07:00:24Z

...rthome.storage.json/src/main/java/org/eclipse/smarthome/storage/json/JsonStorageService.java

+            if (properties.get(CFG_MAX_DEFER_DELAY) != null) {
+                maxDeferredPeriod = Integer.parseInt((String) properties.get(CFG_MAX_DEFER_DELAY));
+            }
+        } catch (NumberFormatException nfe) {


Better do a catch for every single value - as it is now, you will ignore write_delay and defer_delay parameters, if backup_files cannot be parsed, even if they are set correctly.

kaikreuzer · 2016-11-11T07:01:20Z

...rthome.storage.json/src/main/java/org/eclipse/smarthome/storage/json/JsonStorageService.java

+                maxDeferredPeriod = Integer.parseInt((String) properties.get(CFG_MAX_DEFER_DELAY));
+            }
+        } catch (NumberFormatException nfe) {
+            logger.error("NumberFormatException: {}", nfe.getMessage());


Better do a warning like: "Value xxx is invalid for parameter yyy - using zzz as a default instead"

Signed-off-by: Chris Jackson <chris@cd-jackson.com> Also-By: IVAN GEORGIEV ILIEV <ivan.iliev@musala.com>

cdjackson · 2016-11-11T20:51:44Z

Logging updated...

kaikreuzer · 2016-11-11T20:55:10Z

Thanks for your patience :-)
Ok, I will now create a CQ and ask for approval by the IP team. So please refrain from doing any further updates to this branch!

kaikreuzer · 2016-11-11T21:00:03Z

Created CQ 12251.

maggu2810 · 2016-11-23T17:44:22Z

CQ has been approved.

kaikreuzer · 2016-11-23T18:18:48Z

@cdjackson Travis fails with a compilation error on the JSONStorage tests - could you please check? If it needs a fix, please add it as a new commit to your branch (do not squash!) - thanks!

cdjackson · 2016-11-26T09:52:50Z

I've updated this to remove the benchmark 'test' which was causing the issue. I don't think it's a good idea to have it - certainly not as a test anyway.

kaikreuzer · 2016-11-26T10:30:57Z

Thanks @cdjackson, but you forgot to sign-off your second commit!

Signed-off-by: Chris Jackson <chris@cd-jackson.com>

cdjackson · 2016-11-26T10:33:16Z

Sorry - updated...

kaikreuzer · 2016-11-26T10:43:19Z

Perfect, thanks!

maggu2810 · 2016-11-26T18:24:33Z

Isn't it still the wrong license header?

Closes openhab#202 (openhab#202) Sources of info: a) https://community.openhab.org/t/replacement-of-mapdb-by-jsondb/17498 b) eclipse-archived/smarthome#1868 Signed-off-by: Angelos <agf@itcs.ro> (github: AngelosF)

* Add content to jsondb.md Closes #202 (#202) Sources of info: a) https://community.openhab.org/t/replacement-of-mapdb-by-jsondb/17498 b) eclipse-archived/smarthome#1868 Signed-off-by: Angelos <agf@itcs.ro> (github: AngelosF) * Add minor change for testing purpose Signed-off-by: Thomas Dietrich <thomas.dietrich@tu-ilmenau.de> (github: ThomDietrich) * Adressed some review comments Signed-off-by: Jerome Luckenbach <github@luckenba.ch> * Changed to one sentence per line everywhere Signed-off-by: Jerome Luckenbach <github@luckenba.ch> * Improved backup file description Signed-off-by: Jerome Luckenbach <github@luckenba.ch> * Removed empty example file code fences. Signed-off-by: Confectrician <github@luckenba.ch>

watou reviewed Jul 16, 2016
View reviewed changes

kaikreuzer mentioned this pull request Sep 13, 2016

Add step by step instructions and lessons learned to Migration from 1.x tutorial openhab/openhab-docs#86

Closed

maggu2810 reviewed Sep 13, 2016
View reviewed changes

kaikreuzer suggested changes Nov 7, 2016

View reviewed changes

cdjackson force-pushed the json-storage branch 3 times, most recently from 9e3bc10 to 9ee4393 Compare November 8, 2016 17:51

kaikreuzer suggested changes Nov 9, 2016

View reviewed changes

kaikreuzer reviewed Nov 9, 2016

View reviewed changes

cdjackson force-pushed the json-storage branch 3 times, most recently from ce28420 to 5061265 Compare November 9, 2016 19:00

kaikreuzer suggested changes Nov 10, 2016

View reviewed changes

cdjackson force-pushed the json-storage branch from 5061265 to 0b74172 Compare November 10, 2016 16:53

kaikreuzer suggested changes Nov 11, 2016

View reviewed changes

Json Storage Service

9ba4c66

Signed-off-by: Chris Jackson <chris@cd-jackson.com> Also-By: IVAN GEORGIEV ILIEV <ivan.iliev@musala.com>

cdjackson force-pushed the json-storage branch from 0b74172 to 9ba4c66 Compare November 11, 2016 20:48

kaikreuzer added the CQ label Nov 11, 2016

kaikreuzer approved these changes Nov 11, 2016

View reviewed changes

kaikreuzer removed the CQ label Nov 24, 2016

Remove benchmark from tests

d98b79c

Signed-off-by: Chris Jackson <chris@cd-jackson.com>

cdjackson force-pushed the json-storage branch from f402484 to d98b79c Compare November 26, 2016 10:32

kaikreuzer merged commit 83643ef into eclipse-archived:master Nov 26, 2016

cdjackson deleted the json-storage branch December 4, 2016 15:06

AngelosF mentioned this pull request Jan 31, 2017

Add content to jsondb.md openhab/openhab-docs#281

Merged

kaikreuzer mentioned this pull request Feb 16, 2017

JsonDB: Things and Links got missing after Hard Reset #3016

Closed

kaikreuzer added this to the 0.9.0 milestone Nov 30, 2017

		@@ -0,0 +1,129 @@
		package org.eclipse.smarthome.storage.json;


		private final Logger logger = LoggerFactory.getLogger(JsonStorage.class);

		private final int MAX_FILES = 5;

		@@ -0,0 +1,100 @@
		package org.eclipse.smarthome.storage.mapdb;


		private final Map<String, JsonStorage<Object>> storageList = new HashMap<String, JsonStorage<Object>>();

		public void activate() {

JsonDB storage service #1868

JsonDB storage service #1868

Conversation

cdjackson commented Jul 15, 2016

maggu2810 commented Jul 16, 2016

hakan42 commented Jul 16, 2016

hfanieng commented Jul 16, 2016 • edited Loading

cdjackson commented Jul 16, 2016

cdjackson commented Jul 16, 2016

Choose a reason for hiding this comment

smar000 commented Jul 16, 2016

maggu2810 commented Jul 16, 2016

cdjackson commented Jul 16, 2016

thesebastianf commented Jul 31, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kaikreuzer left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cdjackson commented Nov 8, 2016

kaikreuzer left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cdjackson commented Nov 9, 2016

kaikreuzer commented Nov 9, 2016

Choose a reason for hiding this comment

cdjackson commented Nov 10, 2016

kaikreuzer left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cdjackson commented Nov 10, 2016

kaikreuzer left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cdjackson commented Nov 11, 2016

kaikreuzer commented Nov 11, 2016

kaikreuzer commented Nov 11, 2016

maggu2810 commented Nov 23, 2016

kaikreuzer commented Nov 23, 2016

cdjackson commented Nov 26, 2016

kaikreuzer commented Nov 26, 2016

cdjackson commented Nov 26, 2016

kaikreuzer commented Nov 26, 2016

maggu2810 commented Nov 26, 2016

hfanieng commented Jul 16, 2016 •

edited

Loading

kaikreuzer left a comment •

edited

Loading