New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Clear stale records #755

Closed
GiacomoManzoli opened this Issue May 11, 2018 · 5 comments

Comments

Projects
None yet
2 participants
@GiacomoManzoli

GiacomoManzoli commented May 11, 2018

After an error during the check-in procedure on a large IFC file, the BIMserver (v.1.5.96) database got dirty and now the start-up it's slower since it has to check for stale records:

S: Mac OS X
Main class: org.bimserver.JarBimServer

Commands:
java
-Xmx8589m
-Xss1024k
-Dhttp.proxyHost=
-Dhttp.proxyPort=1080
-Dorg.apache.cxf.Logger=org.apache.cxf.common.logging.Slf4jLogger
-classpath .....
address=localhost
port=8082
homedir=/Users/giacomo/Work/bimserverjar/server1596/home
java version "1.8.0_121"
Java(TM) SE Runtime Environment (build 1.8.0_121-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.121-b13, mixed mode)
10:04:59 INFO  [main]: Starting BIMserver                                                               (BimServer.java:232) 
10:04:59 INFO  [main]: Using "/Users/giacomo/Work/bimserverjar/server1596/home" as homedir              (BimServer.java:234) 
10:05:02 INFO  [main]: Adding ~/.m2 as repository                                                       (MavenPluginRepository.java:80) 
Logging to /Users/giacomo/Work/bimserverjar/server1596/home/logs/bimserver.log
10:05:02 INFO  [main]: Version: 1.5.96                                                                  (BimServer.java:325) 
10:05:06 INFO  [main]: Using "Users/giacomo/Work/bimserverjar/server1596/bimserverjar-1.5.96/./emailtemplates/" as template dir (TemplateEngine.java:64) 
10:05:06 INFO  [main]: Non-empty database directory found "/Users/giacomo/Work/bimserverjar/server1596/home/database" (BerkeleyKeyValueStore.java:85) 
10:05:13 INFO  [main]: Adding ~/.m2/repository as repository                                            (MavenPluginRepository.java:80) 
10:06:29 INFO  [main]: Checking for stale records took 73668 ms                                         (BimServer.java:785) 
10:06:29 INFO  [main]: Changing server state to RUNNING                                                 (ServerInfoManager.java:96) 
10:06:31 INFO  [main]: Using exe/64/osx/IfcGeomServer                                                   (IfcOpenShellEnginePlugin.java:129) No expanding necessary

This check it's performed ad each start-up and it's becoming a performance problem, since it's seems that sometimes these records aren't deleted.
Is there a way to manually delete these record or to speed up this process?

@rubendel

This comment has been minimized.

Member

rubendel commented May 14, 2018

Hmm this should indeed not take this long. I don't think this has something to do with the failed checkin, this process simply gets slower and slower with a growing database.

I don't have time to look at this issue right now though, but I'll leave a more technical comment for future me or someone else to fix it:

The nature of this problem is the fact that record keys are composed of pid + oid + rid. We know the pid and also the rid, but the database has no way to "query" for those two specifically (since only the whole key is "indexed"). The only way is to iterate over all records starting with "pid" and skip all the records with the wrong "rid".

The way to solve this is probably to set a bit on the Project object when a checkin starts (and commit it) and then reset it when the checkin has either failed or succeeded. This way only those projects have to be checked. The cleanup process should obviously reset this bit, so that when another restart happens the projects won't be checked a second time. When this is implemented, the "checkinsInProgress" field on the BimServer class can probably be removed and replaced by this bit as well.

rubendel added a commit that referenced this issue Jun 11, 2018

Checkins in progress now simply store the topicId in the Project object.
Since Project objects belong to the "Store" schema, they are stored
transactionally, so the BDB transaction system takes care of concurrent
access.

On BIMserver startup this field (Project.chekinInProgress) is now used
to determine which projects should be cleaned-up, this will be much
faster than before since only projects that had checkins going on need
to be checked. Previously this was stored in memory. One advantage of
this method is that when bugs in BIMserver disallow checkin-in on a
project, restarting the server can (temporarely) solve the problem
(until the bug is fixed).

#755
@rubendel

This comment has been minimized.

Member

rubendel commented Jun 11, 2018

A fix for this has been implemented now and will be in 1.5.102. It's a bit experimental, any feedback is appreciated.

@GiacomoManzoli

This comment has been minimized.

GiacomoManzoli commented Jun 20, 2018

Sorry for the late reply, when it will be released? I don't have enought experience with the Java Build System to build the jar by myself.

@rubendel

This comment has been minimized.

Member

rubendel commented Jun 20, 2018

I hope soon, build system has been migrated and is not working yet, I'd say end of next week

@rubendel

This comment has been minimized.

Member

rubendel commented Sep 24, 2018

This is fixed in the newest builds

@rubendel rubendel closed this Sep 24, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment