Added support for Bdb Space stats. Fixes issue 169 #74

Closed
wants to merge 6 commits into
from

Conversation

Projects
None yet
2 participants
Contributor

ctasada commented Apr 23, 2012

Code based in com.sleepycat.je.util.DbSpace.

Solves issue 169: http://code.google.com/p/project-voldemort/issues/detail?id=169

Collaborator

vinothchandar commented Apr 23, 2012

I am very interested in this. Can we collapse them into one Mbean, i.e simply enhance BdbEnvironmentStats?

Contributor

ctasada commented Apr 24, 2012

Let me check. I'll come back to you with an alternative.

Contributor

ctasada commented Apr 30, 2012

I finally found the time to merge and test the changes. Now everything is in a single MBean.
Just as a note, the "-DJEMonitor=true" env variable must be specified in the voldemort java options.

@vinothchandar vinothchandar and 1 other commented on an outdated diff May 4, 2012

...va/voldemort/store/bdb/stats/BdbEnvironmentStats.java
@@ -34,6 +45,33 @@ private EnvironmentStats getEnvironmentStats(boolean fast) {
return environment.getStats(config);
}
+ private DbSpace getBdbStats() {
+ return new DbSpace(environment, sorted);
@vinothchandar

vinothchandar May 4, 2012

Collaborator

Is nt sorted always false here? can you remove the variable?

@ctasada

ctasada May 4, 2012

Contributor

Done

@vinothchandar vinothchandar and 1 other commented on an outdated diff May 4, 2012

...va/voldemort/store/bdb/stats/BdbEnvironmentStats.java
+ } catch(Exception e) {
+ throw new VoldemortException(e);
+ }
+ }
+
+ @JmxGetter(name = "BdbTotalSize")
+ public long getBdbTotalSize() {
+ return getFastDbStats().getTotal().totalSize();
+ }
+
+ @JmxGetter(name = "BdbTotalUsage")
+ public long getBdbTotalUsage() {
+ return getFastDbStats().getTotal().utilization();
+ }
+
+ @JmxOperation(description = "Displays the disk space utilization for an environment.")
@vinothchandar

vinothchandar May 4, 2012

Collaborator

Why is this a JMXOperation vs JMXGetter? Please make these changes and I will pull this in.

@ctasada

ctasada May 4, 2012

Contributor

Done

Collaborator

vinothchandar commented May 4, 2012

Looks like you forgot to import DbSpace? Our tests broke.

[javac] Compiling 378 source files to /mnt/hudsondisk/hudson_home/jobs/github-pull-requests/workspace/dist/classes
[javac] /mnt/hudsondisk/hudson_home/jobs/github-pull-requests/workspace/src/java/voldemort/store/bdb/stats/BdbEnvironmentStats.java:20: cannot find symbol
[javac] symbol : class DbSpace
[javac] location: class voldemort.store.bdb.stats.BdbEnvironmentStats
[javac] private final CachedCallable fastDbStats;
[javac] ^
[javac] /mnt/hudsondisk/hudson_home/jobs/github-pull-requests/workspace/src/java/voldemort/store/bdb/stats/BdbEnvironmentStats.java:48: cannot find symbol
[javac] symbol : class DbSpace
[javac] location: class voldemort.store.bdb.stats.BdbEnvironmentStats
[javac] private DbSpace getBdbStats() {
[javac] ^
[javac] /mnt/hudsondisk/hudson_home/jobs/github-pull-requests/workspace/src/java/voldemort/store/bdb/stats/BdbEnvironmentStats.java:52: cannot find symbol
[javac] symbol : class DbSpace
[javac] location: class voldemort.store.bdb.stats.BdbEnvironmentStats
[javac] private DbSpace getFastDbStats() {
[javac] ^
[javac] /mnt/hudsondisk/hudson_home/jobs/github-pull-requests/workspace/src/java/voldemort/store/bdb/stats/BdbEnvironmentStats.java:33: cannot find symbol
[javac] symbol : class DbSpace
[javac] location: class voldemort.store.bdb.stats.BdbEnvironmentStats
[javac] Callable fastDbStatsCallable = new Callable() {
[javac] ^
[javac] /mnt/hudsondisk/hudson_home/jobs/github-pull-requests/workspace/src/java/voldemort/store/bdb/stats/BdbEnvironmentStats.java:33: cannot find symbol
[javac] symbol : class DbSpace
[javac] location: class voldemort.store.bdb.stats.BdbEnvironmentStats
[javac] Callable fastDbStatsCallable = new Callable() {
[javac] ^
[javac] /mnt/hudsondisk/hudson_home/jobs/github-pull-requests/workspace/src/java/voldemort/store/bdb/stats/BdbEnvironmentStats.java:35: cannot find symbol
[javac] symbol: class DbSpace
[javac] public DbSpace call() throws Exception {
[javac] ^
[javac] /mnt/hudsondisk/hudson_home/jobs/github-pull-requests/workspace/src/java/voldemort/store/bdb/stats/BdbEnvironmentStats.java:39: cannot find symbol
[javac] symbol : class DbSpace
[javac] location: class voldemort.store.bdb.stats.BdbEnvironmentStats
[javac] fastDbStats = new CachedCallable(fastDbStatsCallable, ttlMs);
[javac] ^
[javac] /mnt/hudsondisk/hudson_home/jobs/github-pull-requests/workspace/src/java/voldemort/store/bdb/stats/BdbEnvironmentStats.java:49: cannot find symbol
[javac] symbol : class DbSpace
[javac] location: class voldemort.store.bdb.stats.BdbEnvironmentStats
[javac] return new DbSpace(environment, sorted);
[javac] ^
[javac] 8 errors

Contributor

ctasada commented May 4, 2012

I forgot to commit the DbSpace class. My fault. I've also done the other requested changes.

@vinothchandar vinothchandar and 1 other commented on an outdated diff May 4, 2012

src/java/voldemort/store/bdb/stats/DbSpace.java
+package voldemort.store.bdb.stats;
+
+import java.util.Iterator;
+import java.util.Map;
+import java.util.SortedMap;
+
+import com.sleepycat.je.DbInternal;
+import com.sleepycat.je.Environment;
+import com.sleepycat.je.cleaner.FileSummary;
+import com.sleepycat.je.cleaner.UtilizationProfile;
+import com.sleepycat.je.dbi.EnvironmentImpl;
+import com.sleepycat.je.log.UtilizationFileReader;
+
+public class DbSpace {
+
+ private boolean recalc = false;
@vinothchandar

vinothchandar May 4, 2012

Collaborator

When is this flag 'true'?

@ctasada

ctasada May 5, 2012

Contributor

The whole code is based in the com.sleepycat.je.util.DbSpace, so some of the command-line arguments came with the code. I've modified the class to accept it as a parameter, because I think can there may be cases were may be needed to recalculate the file summaries

@vinothchandar

vinothchandar May 7, 2012

Collaborator

ya figured that out. But if we are not using it, then we should not have it in there. So, please make those changes as well, either way is fine with me :) -- using it or removing it.

@ctasada

ctasada May 8, 2012

Contributor

Done. I finally decided to remove the recalculation code. It was introducing some unneeded complexity right now. We can always add it later if people demands it :)

@vinothchandar vinothchandar and 1 other commented on an outdated diff May 4, 2012

src/java/voldemort/store/bdb/stats/DbSpace.java
+
+ totals = new Summary();
+ summaries = new Summary[map.size()];
+
+ Iterator<Map.Entry<Long, FileSummary>> iter = map.entrySet().iterator();
+ while(iter.hasNext()) {
+ Map.Entry<Long, FileSummary> entry = iter.next();
+ Long fileNum = entry.getKey();
+ FileSummary fs = entry.getValue();
+ FileSummary recalcFs = null;
+ if(recalcMap != null) {
+ recalcFs = recalcMap.get(fileNum);
+ }
+
+ Summary summary = new Summary(fileNum, fs, recalcFs);
+ if(summaries != null) {
@vinothchandar

vinothchandar May 4, 2012

Collaborator

Why would summaries by null when you "new" it up above

@ctasada

ctasada May 5, 2012

Contributor

I'll clean it up

Collaborator

vinothchandar commented May 8, 2012

Pulled this into my repo. Will be merged as a part of https://github.com/vinothchandar/voldemort/tree/bdb4_1_17

Collaborator

vinothchandar commented May 8, 2012

vinothchandar/voldemort@master...bdb4_1_17 your changes are here. will commit them after perf testing. Closing this

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment