Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP

Loading…

initial integration with MTJ #2

Closed
wants to merge 29 commits into from

2 participants

@fommil

not really for pulling at this stage, but it sort of works and I have a few comments and questions:

  1. the gc root stuff / stack sequences etc confuses me, can you please explain it to me?
  2. I'm getting a lot of undefined references when adding the links (these might be the gc roots, or other artifacts [sic] of a partial dump)
  3. I might have to add a few features to the LinkedSparseMatrix, but for now there is a subclass for playing with (e.g. for doing iteration of just a row or column... i.e. references and referers respectively)
  4. I've avoided using the existing visitors as they rely on the heavyweight Snapshot structures and I wanted to come up with the minimal working example to see how much the memory overhead could be reduced. I think it would be really cool to come up with a nice clean set of base case classes (indexed by value classes so we never have to think it terms of Longs/Ints) that are entirely immutable and hold no references to lookups. This would allow us to separate out the data objects from the storage mechanism.
@rorygraves
Owner

1) will review and come up with some better feedback about them
2) The heapdumps always appear to have unresolved references in them - it appears to be things actual lot written out by the jvm when it dumps the heap. Its very frustrating as it breaks things like graph traversals (e.g. assuming all objects have a root)
3) Cool
4) Yep, thats fair enough the snapshot implementation is based heavily on a tidied up jhat implementation - I don't think its the final form for any of this. Agreed value classes is the way to go.

@fommil

full list of all the missing references in the test file (with number of missing occurrences). A lot of Strings and Classes:

(490,java.lang.reflect.Method.name)
(343,java.lang.String[])
(248,java.lang.Class[])
(243,java.lang.reflect.Method.returnType)
(235,java.lang.reflect.Field.clazz)
(229,java.lang.reflect.Field.name)
(223,java.lang.String.value)
(173,java.util.Hashtable$Entry.key)
(158,java.util.jar.Attributes$Name.name)
(145,java.util.Hashtable$Entry.value)
(138,java.util.LinkedHashMap$Entry.key)
(124,java.net.URL.protocol)
(97,java.util.HashMap$Entry.key)
(92,java.security.Provider$ServiceKey.algorithm)
(90,java.lang.reflect.Field.type)
(80,java.util.HashMap$Entry.value)
(76,java.security.Provider$ServiceKey.type)
(73,java.util.WeakHashMap$Entry.referent)
(58,java.lang.reflect.Constructor.clazz)
(57,javax.management.MBeanAttributeInfo.attributeType)
(48,java.security.Provider$Service.algorithm)
(46,java.security.Provider$UString.string)
(41,java.io.ObjectStreamField.signature)
(40,javax.management.MBeanInfo.className)
(39,java.io.ObjectStreamField.name)
(36,java.security.Provider$Service.type)
(30,java.util.concurrent.ConcurrentHashMap$HashEntry.key)
(29,java.security.Provider$EngineDescription.name)
(26,javax.management.openmbean.OpenMBeanAttributeInfoSupport.attributeType)
(24,java.io.ObjectStreamField.type)
(23,sun.security.x509.OIDMap$OIDInfo.className)
(21,sun.security.x509.AVAKeyword.keyword)
(20,javax.management.MBeanOperationInfo.name)
(18,java.util.Locale.language)
(16,javax.management.MBeanOperationInfo.type)
(14,javax.management.openmbean.SimpleType.typeName)
(11,com.sun.jmx.mbeanserver.PerInterface.mbeanInterface)
(10,javax.management.ObjectName._canonicalName)
(9,java.lang.reflect.Method.annotations)
(8,java.util.logging.LogManager$LoggerWeakRef.name)
(7,javax.management.openmbean.CompositeType.description)
(6,sun.security.x509.OIDMap$OIDInfo.clazz)
(5,sun.security.jca.ServiceId.type)
(4,java.lang.RuntimePermission.name)
(3,sun.security.jca.ServiceId.algorithm)
(2,sun.security.provider.Sun.name)
(1,sun.nio.cs.UTF_8.name)
@fommil

creating the links is ridiculously slow, the data structure needs a rethink on the creation side. My 200MB IntelliJ dump has about 4 million links and creating the structure takes about 20 minutes on my iMac. Hmm.

@fommil fommil disappointingly boring hashmap link structure... could easily be pers…
…isted. also Trove (added) might reduce memory footprint even further.
a29dbea
@rorygraves
Owner

That seems surprising slow - the basic data-structure for the example dump takes a few seconds. Is this created the LinkedSparseMatrix?

@fommil

and the full list of missing heap objects from a 200MB IntelliJ dump:

19:34:19.690 [main] INFO  o.shelmet.heap.parser.LinkingVisitor - 5600000 links
19:34:23.742 [main] INFO  o.s.heap.parser.SparseLinkStructure$ - LINKING pass took 16.78 min
19:34:23.745 [main] INFO  o.s.heap.parser.SparseLinkStructure$ - 2465754 refs with 5621193 links
19:34:23.760 [main] INFO  o.s.heap.parser.SparseLinkStructure$ - (31659,java.lang.Object[])
(14121,org.jdom.Attribute.name)
(10074,org.jdom.Element.name)
(8366,java.lang.reflect.Constructor.clazz)
(7267,java.lang.reflect.Method.name)
(6738,java.lang.reflect.Method.clazz)
(6109,org.jetbrains.plugins.scala.lang.psi.impl.base.ScStableCodeReferenceElementImpl.DEFAULT_ERROR_MESSAGE)
(4304,com.intellij.codeInspection.ex.ScopeToolState.myScopeName)
(4187,org.picocontainer.defaults.ConstructorInjectionComponentAdapter.componentImplementation)
(4105,org.jetbrains.plugins.scala.lang.psi.impl.base.ScModifierListImpl.DEFAULT_ERROR_MESSAGE)
(4023,java.lang.reflect.Field.clazz)
(3894,java.lang.reflect.Field.name)
(3824,org.jetbrains.plugins.scala.lang.psi.impl.base.types.ScSimpleTypeElementImpl.DEFAULT_ERROR_MESSAGE)
(3704,java.lang.String.value)
(3607,com.intellij.openapi.extensions.impl.ExtensionComponentAdapter.myImplementationClass)
(3566,org.jetbrains.plugins.scala.lang.psi.impl.expr.ScAnnotationsImpl.DEFAULT_ERROR_MESSAGE)
(3339,java.lang.Class[])
(3081,java.lang.String[])
(2882,org.jetbrains.plugins.scala.lang.psi.impl.statements.params.ScParametersImpl.DEFAULT_ERROR_MESSAGE)
(2847,org.jetbrains.plugins.scala.lang.psi.impl.statements.params.ScParameterImpl.DEFAULT_ERROR_MESSAGE)
(2656,java.util.LinkedHashMap$Entry.key)
(2477,com.intellij.util.containers.ConcurrentHashMap$Node.key)
(2369,org.jetbrains.plugins.scala.lang.psi.impl.statements.ScFunctionDefinitionImpl.DEFAULT_ERROR_MESSAGE)
(2366,org.jetbrains.plugins.scala.lang.psi.impl.statements.params.ScParameterClauseImpl.DEFAULT_ERROR_MESSAGE)
(2356,java.lang.reflect.Method.returnType)
(2143,com.intellij.testFramework.LightVirtualFile.myName)
(2056,java.util.HashMap$Entry.key)
(1893,java.util.EnumMap.keyType)
(1561,com.intellij.patterns.ObjectPattern$1.myAcceptedClass)
(1389,org.jetbrains.plugins.scala.lang.psi.impl.statements.params.ScTypeParamImpl.DEFAULT_ERROR_MESSAGE)
(1242,com.intellij.patterns.TreeElementPattern$4.myDebugMethodName)
(1194,java.util.Hashtable$Entry.key)
(1136,org.jetbrains.plugins.scala.lang.psi.impl.toplevel.synthetic.ScSyntheticFunction.name)
(1135,org.jetbrains.plugins.scala.lang.psi.impl.base.types.ScParameterizedTypeElementImpl.DEFAULT_ERROR_MESSAGE)
(1104,org.jetbrains.plugins.scala.lang.psi.impl.base.types.ScTypeArgsImpl.DEFAULT_ERROR_MESSAGE)
(970,org.jetbrains.plugins.scala.lang.psi.impl.statements.params.ScTypeParamClauseImpl.DEFAULT_ERROR_MESSAGE)
(843,sun.reflect.annotation.AnnotationInvocationHandler.type)
(830,org.apache.log4j.Logger.name)
(829,org.apache.log4j.CategoryKey.name)
(731,org.jetbrains.plugins.scala.lang.psi.impl.toplevel.synthetic.ScSyntheticFunction.DEFAULT_ERROR_MESSAGE)
(704,java.lang.reflect.Field.type)
(692,com.intellij.patterns.ObjectPattern$3.myDebugMethodName)
(669,org.jdom.Attribute.value)
(667,com.intellij.openapi.util.Pair.first)
(645,com.intellij.ide.ui.search.OptionDescription.myHit)
(632,java.util.LinkedHashMap$Entry.value)
(619,com.intellij.patterns.ObjectPattern$3.val$o)
(614,com.intellij.util.xmlb.BeanBinding.myBeanClass)
(610,com.intellij.patterns.PsiAnnotationPattern$1.myDebugMethodName)
(582,com.intellij.util.containers.ConcurrentSoftValueHashMap$MySoftReference.key)
(539,org.jetbrains.plugins.scala.lang.psi.impl.base.patterns.ScReferencePatternImpl.DEFAULT_ERROR_MESSAGE)
(536,org.jetbrains.plugins.scala.lang.psi.impl.base.ScPatternListImpl.DEFAULT_ERROR_MESSAGE)
(522,com.intellij.openapi.util.Key.myName)
(514,org.jetbrains.plugins.scala.lang.psi.impl.statements.ScPatternDefinitionImpl.DEFAULT_ERROR_MESSAGE)
(486,scala.reflect.ClassTag$$anon$1.runtimeClass1$1)
(481,com.intellij.psi.impl.source.tree.LeafPsiElement.h)
(474,com.intellij.util.keyFMap.OneElementFMap.myValue)
(463,java.util.HashMap$Entry.value)
(461,org.jetbrains.plugins.scala.lang.psi.impl.toplevel.templates.ScExtendsBlockImpl.DEFAULT_ERROR_MESSAGE)
(435,java.lang.reflect.Type[])
(427,org.jetbrains.plugins.scala.lang.psi.impl.toplevel.templates.ScTemplateBodyImpl.DEFAULT_ERROR_MESSAGE)
(424,org.jetbrains.plugins.scala.lang.psi.impl.statements.ScFunctionDeclarationImpl.DEFAULT_ERROR_MESSAGE)
(423,scala.collection.mutable.DefaultEntry.key)
(420,java.lang.reflect.Field.annotations)
(410,java.util.WeakHashMap$Entry.referent)
(404,org.jetbrains.plugins.scala.lang.psi.impl.toplevel.packaging.ScPackagingImpl.DEFAULT_ERROR_MESSAGE)
(402,com.intellij.util.xmlb.OptionTagBinding.myName)
(383,com.intellij.util.xml.JavaMethodSignature.myMethodName)
(364,java.util.Hashtable$Entry.value)
(340,java.lang.ref.WeakReference.referent)
(303,java.lang.reflect.Method.annotations)
(302,org.jetbrains.plugins.scala.lang.psi.impl.base.ScAccessModifierImpl.DEFAULT_ERROR_MESSAGE)
(297,com.intellij.util.xml.JavaMethod.myDeclaringClass)
(296,java.lang.reflect.Method.annotationDefault)
(265,org.jetbrains.plugins.scala.lang.psi.impl.toplevel.typedef.ScClassImpl.DEFAULT_ERROR_MESSAGE)
(261,org.jetbrains.plugins.scala.lang.psi.impl.expr.ScAnnotationImpl.DEFAULT_ERROR_MESSAGE)
(260,com.intellij.openapi.extensions.impl.ExtensionPointImpl.myExtensionClass)
(259,java.io.ObjectStreamField.signature)
(253,org.jetbrains.plugins.scala.lang.psi.impl.toplevel.typedef.ScObjectImpl.DEFAULT_ERROR_MESSAGE)
(251,sun.reflect.generics.reflectiveObjects.ParameterizedTypeImpl.rawType)
(250,com.intellij.patterns.PsiNameValuePairPattern$1.myDebugMethodName)
(245,org.jetbrains.plugins.scala.lang.psi.impl.toplevel.typedef.ScTraitImpl.DEFAULT_ERROR_MESSAGE)
(234,com.intellij.openapi.extensions.ExtensionPointName.myName)
(229,org.jetbrains.plugins.scala.lang.psi.impl.base.ScConstructorImpl.DEFAULT_ERROR_MESSAGE)
(225,java.util.ResourceBundle$CacheKey.format)
(223,org.jetbrains.plugins.scala.lang.psi.impl.statements.ScTypeAliasDefinitionImpl.DEFAULT_ERROR_MESSAGE)
(221,org.jetbrains.plugins.scala.lang.psi.impl.toplevel.templates.ScClassParentsImpl.DEFAULT_ERROR_MESSAGE)
(205,org.jetbrains.plugins.scala.lang.psi.impl.toplevel.templates.ScTraitParentsImpl.DEFAULT_ERROR_MESSAGE)
(204,com.intellij.javaee.ExternalResourceManagerExImpl$Resource.clazz)
(203,com.intellij.plugins.drools.lang.lexer.DroolsElementType.myDebugName)
(195,java.io.ObjectStreamField.name)
(187,sun.reflect.generics.scope.ClassScope.recvr)
(176,com.intellij.psi.impl.source.tree.PsiWhiteSpaceImpl.h)
(173,org.jetbrains.plugins.scala.lang.lexer.ScalaElementType.myDebugName)
(163,com.intellij.lang.javascript.JSElementType.myDebugName)
(157,java.util.regex.Pattern.pattern)
(153,com.intellij.patterns.PsiNameValuePairPattern$1.val$requiredName)
(133,java.util.concurrent.ConcurrentHashMap$HashEntry.key)
(131,com.intellij.openapi.editor.colors.TextAttributesKey.myExternalName)
(130,org.jetbrains.plugins.scala.lang.psi.impl.expr.ScArgumentExprListImpl.DEFAULT_ERROR_MESSAGE)
(122,com.intellij.openapi.actionSystem.DataKey.myName)
(118,java.security.Provider$ServiceKey.algorithm)
(117,com.intellij.jpa.ql.psi.QlTokenType.myDebugName)
(114,org.jetbrains.plugins.scala.lang.psi.impl.statements.params.ScParameterTypeImpl.DEFAULT_ERROR_MESSAGE)
(106,org.jetbrains.plugins.scala.lang.psi.impl.base.ScPrimaryConstructorImpl.DEFAULT_ERROR_MESSAGE)
(104,org.jetbrains.plugins.scala.lang.psi.impl.statements.params.ScClassParameterImpl.DEFAULT_ERROR_MESSAGE)
(100,org.jetbrains.plugins.scala.lang.psi.impl.expr.ScMethodCallImpl.DEFAULT_ERROR_MESSAGE)
(98,com.intellij.psi.ref.AnnotationChildLink.myAnnoFqn)
(96,com.intellij.patterns.PsiModifierListOwnerPattern$3.myDebugMethodName)
(92,com.intellij.patterns.PsiModifierListOwnerPattern$3.val$qualifiedName)
(85,com.intellij.util.containers.ConcurrentHashMap$Node.val)
(84,org.jetbrains.plugins.scala.lang.psi.impl.expr.ScInfixExprImpl.DEFAULT_ERROR_MESSAGE)
(82,com.intellij.psi.codeStyle.CodeStyleSettingsCustomizable$WrappingOrBraceOption.name)
(80,com.intellij.util.xmlb.PropertyAccessor.myType)
(79,org.jetbrains.plugins.scala.lang.psi.impl.base.ScLiteralImpl.DEFAULT_ERROR_MESSAGE)
(76,java.beans.MethodDescriptor.name)
(74,org.apache.xerces.util.SymbolTable$Entry.symbol)
(70,java.io.ObjectStreamClass$WeakClassKey.referent)
(69,com.intellij.patterns.ObjectPattern$4.myDebugMethodName)
(68,scala.collection.immutable.HashSet$HashSet1.key)
(67,org.intellij.lang.regexp.RegExpElementType.myDebugName)
(66,com.intellij.psi.codeStyle.CodeStyleSettingsCustomizable$SpacingOption.name)
(64,com.intellij.util.keyFMap.PairElementsFMap.value1)
(63,java.util.jar.Attributes$Name.name)
(62,sun.security.pkcs11.SunPKCS11$Descriptor.algorithm)
(61,com.intellij.psi.tree.xml.IXmlLeafElementType.myDebugName)
(60,com.intellij.patterns.XmlAttributeValuePattern$2.myDebugMethodName)
(59,com.intellij.psi.ref.AnnotationAttributeChildLink.myAttributeName)
(58,java.io.ObjectStreamClass.name)
(57,com.intellij.psi.tree.java.IJavaElementType.myDebugName)
(56,java.security.Provider$Service.algorithm)
(55,com.intellij.tasks.jira.jql.JqlElementType.myClass)
(54,com.intellij.tasks.jira.jql.JqlElementType.myDebugName)
(53,com.intellij.psi.tree.java.IKeywordElementType.myDebugName)
(52,com.sun.tools.javac.main.OptionName.name)
(51,com.intellij.util.messages.Topic.myDisplayName)
(50,com.intellij.patterns.PsiJavaPatterns$4.myAcceptedClass)
(49,javax.swing.text.html.CSS$Attribute.name)
(48,org.jetbrains.plugins.scala.lang.psi.impl.expr.ScBlockExprImpl.DEFAULT_ERROR_MESSAGE)
(46,com.intellij.patterns.PsiElementPattern$5.myDebugMethodName)
(45,com.intellij.jam.JamClassGeneratorImpl$13.val$base)
(42,com.intellij.util.xml.impl.CollectionChildDescriptionImpl.a)
(41,com.intellij.psi.impl.smartPointers.SelfElementInfo.d)
(40,javax.management.MBeanInfo.className)
(39,java.io.ObjectStreamClass$FieldReflectorKey.referent)
(38,javax.swing.text.html.CSS$Attribute.defaultValue)
(36,scala.Tuple2._1)
(34,sun.swing.SwingLazyValue.methodName)
(32,org.jetbrains.plugins.scala.lang.scaladoc.lexer.ScalaDocElementType.myDebugName)
(30,com.intellij.util.xmlb.CollectionBinding.myElementType)
(29,javax.accessibility.AccessibleState.defaultResourceBundleName)
(27,io.netty.util.internal.chmv8.ConcurrentHashMapV8$Node.key)
(26,org.jetbrains.plugins.scala.lang.psi.impl.base.types.ScCompoundTypeElementImpl.DEFAULT_ERROR_MESSAGE)
(25,io.netty.channel.ChannelOption.name)
(24,com.intellij.util.xmlb.PrimitiveValueBinding.myType)
(23,com.intellij.patterns.PsiMemberPattern$1.myDebugMethodName)
(22,org.jetbrains.plugins.scala.lang.psi.impl.statements.ScVariableDefinitionImpl.DEFAULT_ERROR_MESSAGE)
(21,com.intellij.psi.tree.IFileElementType.myDebugName)
(20,javax.swing.text.html.CSS$Value.name)
(19,java.util.Locale.language)
(18,com.intellij.psi.tree.jsp.IJspElementType.myDebugName)
(17,java.lang.RuntimePermission.name)
(16,sun.reflect.generics.reflectiveObjects.TypeVariableImpl.genericDeclaration)
(15,org.jetbrains.plugins.scala.lang.psi.impl.toplevel.synthetic.SyntheticClasses$$anon$1.name)
(14,com.intellij.lang.javascript.index.JSNamedElementIndexItemBase$NamedItemType.name)
(13,org.jetbrains.plugins.scala.lang.psi.stubs.impl.ScTemplateDefinitionStubImpl.myJavaName)
(12,com.intellij.lang.javascript.dialects.JSLanguageFeature.myDescr)
(11,com.sun.jmx.remote.util.ClassLogger.className)
(10,com.intellij.openapi.externalSystem.model.Key.myDataClass)
(9,com.intellij.codeInspection.ProblemHighlightType.name)
(8,org.jetbrains.plugins.scala.lang.psi.impl.expr.ScPrefixExprImpl.DEFAULT_ERROR_MESSAGE)
(7,javax.management.openmbean.CompositeType.description)
(6,sun.security.x509.OIDMap$OIDInfo.clazz)
(5,com.siyeh.ig.logging.LogStatementGuardedByLogConditionInspection.loggerClassName)
(4,org.jetbrains.jps.api.CmdlineRemoteProto$Message$ControllerMessage$Type.name)
(3,org.jetbrains.plugins.scala.config.DebuggingInfoLevel.myOption)
(2,com.intellij.ide.favoritesTreeView.AbstractFavoritesListProvider$1.myValue)
(1,org.jetbrains.plugins.scala.codeInsight.intention.format.ConvertFormattedStringToInterpolatedString.myText)
@fommil

BTW, parallel linked matrix structure thoughts here: fommil/matrix-toolkits-java#44

@rorygraves
Owner

Surprising that there are non-core classes in there - I had assumed it was a root classloader/core class type problem.

@fommil

Hmm, I wonder if I'm registering the statics correctly...

@fommil fommil closed this
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Commits on Feb 2, 2014
  1. @fommil
  2. @fommil

    disappointingly boring hashmap link structure... could easily be pers…

    fommil authored
    …isted. also Trove (added) might reduce memory footprint even further.
Commits on Feb 3, 2014
  1. @fommil
  2. @fommil

    experimental linked matrix

    fommil authored
  3. @fommil
  4. @fommil
Commits on Feb 4, 2014
  1. @fommil
  2. @fommil

    primitive collections

    fommil authored
Commits on Feb 6, 2014
  1. @fommil
Commits on Feb 7, 2014
  1. @fommil

    use debox instead of trove: linking is now about 3 times the time of …

    fommil authored
    …indexing the classes. Not too shabby, but concurrency in the primitive collections would be even better.
  2. @fommil

    slight speedup

    fommil authored
  3. @fommil

    cleanup debox typesigs

    fommil authored
  4. @fommil

    groundwork for RX

    fommil authored
Commits on Feb 8, 2014
  1. @fommil
  2. @fommil
Commits on Feb 9, 2014
  1. @fommil
  2. @fommil

    linking is down to half the memory of the heap dump and 30 seconds pe…

    fommil authored
    …r GB, but we require two initial passes to create ref and class lookups. This could be persisted.
  3. @fommil

    testcase for visualvm list

    fommil authored
  4. @fommil

    .

    fommil authored
  5. @fommil

    .

    fommil authored
  6. @fommil
  7. @fommil

    .

    fommil authored
Commits on Feb 10, 2014
  1. @fommil

    .

    fommil authored
  2. @fommil

    .

    fommil authored
Commits on Feb 11, 2014
  1. @fommil

    .

    fommil authored
  2. @fommil

    .

    fommil authored
  3. @fommil

    .

    fommil authored
  4. @fommil

    .

    fommil authored
Commits on Feb 12, 2014
  1. @fommil

    .

    fommil authored
This page is out of date. Refresh to see the latest.
View
60 build.sbt
@@ -1,39 +1,39 @@
-import AssemblyKeys._ // put this at the top of the file
-
-assemblySettings
-
name := "shelmet"
-version := "1.0"
+version := "1.0-SNAPSHOT"
scalaVersion := "2.10.3"
//scalacOptions ++= Seq("-Xplugin:/workspace/recode/recode-compiler-plugin/target/scala-2.10/recode-compiler-plugin_2.10-1.0.jar")
-resolvers += "spray repo" at "http://repo.spray.io"
-
-resolvers += "Typesafe Repository" at "http://repo.typesafe.com/typesafe/releases/"
-
-resolvers += Resolver.sonatypeRepo("snapshots")
-
-libraryDependencies += "com.github.scopt" %% "scopt" % "3.2.0"
-
-libraryDependencies += "ch.qos.logback" % "logback-classic" % "1.0.13"
-
-libraryDependencies += "org.scalatest" %% "scalatest" % "2.0" % "test"
-
-libraryDependencies += "xmlunit" % "xmlunit" % "1.5" % "test"
-
-libraryDependencies += "com.typesafe" %% "scalalogging-slf4j" % "1.0.1"
-
-libraryDependencies += "io.spray" % "spray-can" % "1.2.0"
-
-libraryDependencies += "io.spray" % "spray-routing" % "1.2.0"
-
-libraryDependencies += "com.typesafe.akka" %% "akka-actor" % "2.2.3"
-
-libraryDependencies += "com.typesafe.akka" % "akka-slf4j_2.10" % "2.2.3"
-
-libraryDependencies += "com.typesafe.akka" %% "akka-testkit" % "2.2.3" % "test"
+resolvers ++= Seq(
+ Resolver.mavenLocal,
+ Resolver.sonatypeRepo("snapshots"),
+ Resolver.typesafeRepo("releases"),
+ "spray repo" at "http://repo.spray.io"
+)
+
+libraryDependencies ++= List(
+ "com.github.scopt" %% "scopt" % "3.2.0",
+ "ch.qos.logback" % "logback-classic" % "1.0.13",
+ "org.scalatest" %% "scalatest" % "2.0" % "test",
+ "xmlunit" % "xmlunit" % "1.5" % "test",
+ "com.typesafe" %% "scalalogging-slf4j" % "1.0.1",
+ "io.spray" % "spray-can" % "1.2.0" exclude("org.scala-lang", "scala-library"),
+ "io.spray" % "spray-routing" % "1.2.0" exclude("org.scala-lang", "scala-library"),
+ "com.googlecode.matrix-toolkits-java" % "mtj" % "1.0.2-SNAPSHOT" intransitive(),
+ "com.github.fommil" %% "debox" % "0.1.1-SNAPSHOT",
+ "com.netflix.rxjava" % "rxjava-scala" % "0.16.1" intransitive(),
+ "com.netflix.rxjava" % "rxjava-core" % "0.16.1" intransitive(),
+ "com.google.guava" % "guava" % "16.0",
+ "org.mapdb" % "mapdb" % "0.9.9",
+// "net.sf.trove4j" % "trove4j" % "3.0.3",
+ "com.typesafe.akka" %% "akka-actor" % "2.2.3",
+ "com.typesafe.akka" % "akka-slf4j_2.10" % "2.2.3",
+ "com.typesafe.akka" %% "akka-testkit" % "2.2.3" % "test",
+ "org.specs2" %% "specs2" % "2.2.3" % "test"
+)
+
+ideaExcludeFolders ++= Seq(".idea", ".idea_modules")
net.virtualvoid.sbt.graph.Plugin.graphSettings
View
1  project/build.properties
@@ -0,0 +1 @@
+sbt.version=0.13.1
View
9 project/plugins.sbt
@@ -1,8 +1,7 @@
-addSbtPlugin("com.github.mpeltonen" % "sbt-idea" % "1.5.2")
-
-addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.10.1")
-
addSbtPlugin("net.virtual-void" % "sbt-dependency-graph" % "0.7.4")
+//addSbtPlugin("com.github.mpeltonen" % "sbt-idea" % "1.5.2")
+
+resolvers += "Sonatype snapshots" at "http://oss.sonatype.org/content/repositories/snapshots/"
-addSbtPlugin("com.typesafe.sbteclipse" % "sbteclipse-plugin" % "2.4.0")
+addSbtPlugin("com.github.mpeltonen" % "sbt-idea" % "1.6.0-SNAPSHOT")
View
11 src/main/scala/org/shelmet/heap/parser/HProfObservable.scala
@@ -0,0 +1,11 @@
+package org.shelmet.heap.parser
+
+import rx.lang.scala.Observable
+
+trait HProfBlock
+
+
+class HProfObservable {
+
+
+}
View
341 src/main/scala/org/shelmet/heap/parser/SparseLinkStructure.scala
@@ -0,0 +1,341 @@
+package org.shelmet.heap.parser
+
+import com.typesafe.scalalogging.slf4j.Logging
+import scala.collection.mutable
+import org.shelmet.heap.shared.{BaseFieldType, FieldType, ClassType}
+import org.shelmet.heap.HeapId
+import com.google.common.base.Stopwatch
+import debox.map.{Map => DMap}
+import java.io.{BufferedInputStream, File, FileInputStream}
+import org.mapdb.{Fun, BTreeKeySerializer, DBMaker}
+import org.mapdb.Fun.Tuple2
+
+object SparseLinkStructure extends App with Logging {
+
+ val file = new File("intellij.hprof")
+
+}
+
+class PersistentStructure(heapDump: File) extends Logging {
+ // TODO: these would work really well as pimps
+
+ def typez(ref: Ref): Typez = ???
+
+ def referencedBy(ref: Ref): Iterable[Ref] = ???
+
+// needs forward lookup
+// def references(ref: Ref): Iterable[Ref] = ???
+
+// needs RAM loader
+// def values(ref: Ref)
+
+
+// logger.info(s"${refs.length } refs (${size(refs)}), ${links.entered} links parsed, (${links.count } unique), ${classes.size } classes and ${refs.length - links.columnSize } refs without metadata")
+
+ private val dbFile = new File(heapDump + ".idx")
+ private def clearDb = {
+ dbFile.delete()
+ new File(dbFile + ".p").delete()
+ new File(dbFile + ".t").delete()
+ Thread.sleep(1000)
+ }
+ private def loadDb = DBMaker.newFileDB(dbFile).
+ asyncWriteEnable.
+ fullChunkAllocationEnable.
+ make()
+
+ private val db = {
+ val trial = loadDb
+ if (trial.getAtomicBoolean("indexed").get) {
+ val ts = trial.getTreeSet[Tuple2[Int, Int]]("linkedby")
+ logger.info("links = " + ts.size())
+ ???
+ } else {
+ logger.info("creating a fresh DB")
+ trial.close()
+ clearDb
+ loadDb
+ }
+ }
+
+ val reader = new HprofReader(file.toString)
+
+ def timed[T <: AnyRef](text: String)(f: => T): T = {
+ val watch = Stopwatch.createStarted()
+ f withEffect { r =>
+ logger.info(text + " took " + watch)
+ }
+ }
+
+ timed("reading full file") {
+ // for benchmarking
+ val reader = new BufferedInputStream(new FileInputStream(file))
+ try {
+ val buffer = Array.ofDim[Byte](1024)
+ while (reader.read(buffer) != -1) {}
+ "done"
+ } finally reader.close()
+ }
+
+ timed("null reader") {
+ reader.readFile(new DumpVisitor {})
+ "done"
+ }
+
+ // memory cost is about 2 * number of instances in the system (plus negligible for classes)
+ val (classes, refs, refTypes, arrays) = timed("class visitor") {
+ val visitor = new ClassVisitor
+ reader.readFile(visitor)
+ (visitor.lookup, visitor.refs, visitor.refTypes, visitor.arrays)
+ }
+
+ logger.info(s"seen ${refs.length } refs and ${arrays.length } arrays")
+
+ val links = timed("linking") {
+ // and mutates "refs"
+ val visitor = new LinkingVisitor(classes, refs)
+ reader.readFile(visitor)
+ visitor.links
+ }
+
+// def size[K,V](map: DMap[K,V]) = {
+// var count = 0
+// map foreach { (k,v) =>
+// count += 1
+// }
+// count
+// }
+
+
+ // FIXME: DMap doesn't report correct length
+
+
+// timed("persisting everything") {
+// db.createTreeMap("classes").makeLongMap[Clazz] withEffect { ts =>
+// classes foreach {c => ts.put(c._1.id, c._2)}
+// logger.info("classes = " + ts.size())
+// }
+// db.commit()
+// logger.info("classes done")
+// db.createTreeMap("refs").makeLongMap[Int] withEffect { ts =>
+// refs foreach {(k,v) => ts.put(k, v)}
+// }
+// db.commit()
+// logger.info("refs done")
+// db.createTreeMap("refType").makeLongMap[Long] withEffect { ts =>
+// refTypes foreach {(k,v) => ts.put(k, v.id)}
+// logger.info("refType = " + ts.size())
+// }
+// db.commit()
+// logger.info("refTypes done")
+// db.createTreeMap("arrays").makeLongMap[Int] withEffect { ts =>
+// arrays foreach {(k,v) => ts.put(k, v)}
+// logger.info("arrays = " + ts.size())
+// }
+// db.commit()
+// logger.info("arrays done")
+// db.createTreeSet("linkedby").serializer(BTreeKeySerializer.TUPLE2).make[Tuple2[Int, Int]] withEffect { ts =>
+// // https://github.com/jankotek/MapDB/blob/master/src/test/java/examples/MultiMap.java
+// // note, we could also be fancy and save a long (64 = 32 * 2)
+// (0 until links.columnSize) foreach { i =>
+// links.get(i) foreach {j =>
+// ts.add(Fun.t2(i, j))
+// }
+// }
+// logger.info("linkedby = " + ts.size())
+// }
+// db.commit()
+// logger.info("linkedby done")
+// db.createAtomicBoolean("indexed", true)
+// db.commit()
+// db.close()
+// "done"
+// }
+}
+
+class PersistingParser {
+
+}
+
+
+class Ref(val id: Int) extends AnyVal {
+ def isNull = id == 0
+}
+
+object Ref {
+ def apply(id: Int): Ref = new Ref(id)
+}
+
+
+class NameId(val id: Long) extends AnyVal
+
+object NameId {
+ def apply(id: Long) = new NameId(id)
+}
+
+class ClassId(val id: Long) extends AnyVal
+
+object ClassId {
+ // def apply(id: Long) = new ClassId(id)
+ def apply(id: HeapId) = new ClassId(id.id)
+}
+
+
+case class Field(name: String, fieldType: FieldType) {
+ def isRef = fieldType.isObjectType
+}
+
+sealed trait Typez
+
+case class Arraz(base: BaseFieldType, length: Int) extends Typez
+
+// ignoring everything I don't care about for now
+// ideally this would be an interim object and the version with the full
+// fields would be passed around after the initial scan
+case class Clazz(name: ClassType, fields: List[Field]) extends Typez {
+ def types = fields map {
+ _.fieldType
+ }
+
+ def fullFieldName(i: Int) = name + "." + fields(i).name
+}
+
+
+trait RefRecorder {
+ // HeapId => Ref
+ def refs: DMap[Long, Int]
+
+ def getRef(id: HeapId) =
+ if (id.isNull) Ref(0)
+ else Ref(refs.get(id.id) match {
+ case None =>
+ val ref = refs.length
+ refs(id.id) = ref
+ ref
+ case Some(ref) => ref
+ })
+}
+
+
+// first pass
+// we only care about creating Refs for the references that have a heap dump segment at this stage
+// as they are the only things we care about for the linking
+class ClassVisitor extends DumpVisitor with RefRecorder with Logging {
+ // relatively small objects, cleanliness of type safety
+ // is better than negligible debox performance improvement
+ private val classNames = mutable.HashMap[ClassId, NameId]()
+ private val names = mutable.HashMap[NameId, String]() withDefault ("unknown@" + _)
+ private val classes = mutable.HashMap[ClassId, LazyClazz]()
+
+ def lookup = classes.mapValues(_.toClazz(classes)).toMap
+
+ val refs = DMap.empty[Long, Int]
+ refs(0) = 0
+
+ // Ref => ClassId
+ val refTypes = DMap.empty[Int, ClassId]
+ // Ref => array size in bytes
+ val arrays = DMap.empty[Int, Int]
+
+ override def utf8String(id: Long, string: String) {
+ names += NameId(id) -> string
+ }
+
+ // presumably all the loadClass calls happen at the start
+ override def loadClass(classSerialNo: Int, classID: HeapId, stackTraceSerialNo: Int, classNameId: Long) {
+ classNames += ClassId(classID) -> NameId(classNameId)
+ //getRef(classID) // classes can be referenced, but is this useful for the linking?
+ }
+
+ case class LazyClazz(name: ClassType, superClass: ClassId, fields: List[Field]) {
+ def allFields(lookup: collection.Map[ClassId, LazyClazz]): List[Field] = fields ::: {
+ lookup.get(superClass) match {
+ case None => Nil
+ case Some(inherited) => inherited.allFields(lookup)
+ }
+ }
+
+ def toClazz(lookup: collection.Map[ClassId, LazyClazz]) = Clazz(name, allFields(lookup))
+ }
+
+ override def classDump(id: HeapId, stackTraceSerialId: Int,
+ superClassId: HeapId,
+ classLoaderId: HeapId,
+ signerId: HeapId,
+ protDomainId: HeapId,
+ instanceSize: Int,
+ constPoolEntries: Map[Int, Any],
+ staticItems: List[ClassStaticEntry],
+ fieldItems: List[ClassFieldEntry]) {
+ val classSig = names(classNames(ClassId(id)))
+ val classType = ClassType.parse(classSig)
+
+ val fields = fieldItems.map { fi =>
+ val fieldName = names(NameId(fi.nameId))
+ Field(fieldName, fi.itemType)
+ }
+ // static items can be referenced, but is this useful for the linking?
+ // const pool are not referenced so are not linked
+ // staticItems foreach {_.value match {
+ // case id: HeapId if !id.isNull => getRef(id)
+ // case _ =>
+ // }
+ // }
+ classes += ClassId(id) -> LazyClazz(classType, ClassId(superClassId), fields)
+ }
+
+ override def instanceDump(id: HeapId, x: Int, classId: HeapId, fields: Option[Vector[Any]], xx: Int) {
+ refTypes(getRef(id).id) = ClassId(classId)
+ }
+
+ override def objectArrayDump(id: HeapId, x: Int, numElements: Int, classId: HeapId, elementIDs: Seq[HeapId]) {
+ arrays(getRef(id).id) = numElements
+ }
+
+ override def primitiveArray(id: HeapId, x: Int, fieldType: BaseFieldType, data: Seq[AnyVal]) {
+ arrays(getRef(id).id) = data.length * fieldType.fieldSize
+ }
+}
+
+
+trait ClassAwareDumpVisitor extends DumpVisitor {
+ def classes: Map[ClassId, Clazz]
+
+ final override def getClassFieldInfo(classHeapId: HeapId) =
+ classes.get(ClassId(classHeapId)) map { c =>
+ for {field <- c.fields} yield field.fieldType
+ }
+}
+
+
+class LinkingVisitor(val classes: Map[ClassId, Clazz], val refs: DMap[Long, Int]) extends ClassAwareDumpVisitor with RefRecorder with Logging {
+
+ val links = new CompressedColumns(refs.length)
+
+ private def link(from: Ref, to: Ref) {
+ if (to.id >= links.columnSize) return
+
+ links.set(from.id, to.id)
+ val tally = links.entered
+ if (tally % 1000000 == 0)
+ logger.info(tally + " links")
+ }
+
+ override def instanceDump(id: HeapId, x: Int, classId: HeapId, fields: Option[Vector[Any]], xx: Int) {
+ val from = getRef(id)
+ fields.get foreach {
+ _ match {
+ case to: HeapId if !to.isNull =>
+ link(from, getRef(to))
+ case _ => // primitive
+ }
+ }
+ }
+
+ override def objectArrayDump(id: HeapId, x: Int, numElements: Int, classId: HeapId, elementIDs: Seq[HeapId]) {
+ val from = getRef(id)
+ elementIDs foreach { to =>
+ if (!to.isNull) link(from, getRef(to))
+ }
+ }
+}
View
231 src/main/scala/org/shelmet/heap/parser/hack.scala
@@ -0,0 +1,231 @@
+package org.shelmet.heap.parser
+
+import java.util.concurrent.atomic.{AtomicReferenceArray, AtomicLong}
+import com.typesafe.scalalogging.slf4j.Logging
+import java.util.concurrent.locks.ReentrantLock
+
+object `package` {
+
+ implicit class PimpedEffects[T <: AnyRef](val a: T) {
+ def withEffect(effect: T => Unit): T = {
+ effect(a)
+ a
+ }
+
+ def synced[R](f: T => R): R = a synchronized {
+ f(a)
+ }
+ }
+
+}
+
+// mutable, but allows concurrent set
+class LinkedMatrix(size: Int) {
+
+ class Node(var row: Int,
+ var col: Int,
+ var value: Double,
+ var rowTail: Node,
+ var colTail: Node)
+
+ // rows and cols store the zeroeth col and row respectively
+ // (0, 0) is arbitrarily stored in "rows"
+ private val cols = Array.ofDim[Node](size)
+ private val rows = Array.ofDim[Node](size)
+
+ rows(size - 1) = new Node(size - 1, 0, 0, null, null)
+ cols(size - 1) = new Node(0, size - 1, 0, null, null)
+ (0 until size - 1).reverse foreach {
+ i =>
+ rows(i) = new Node(i, 0, 0, rows(i + 1), rows(i + 1))
+ cols(i) = new Node(0, i, 0, cols(i + 1), cols(i + 1))
+ }
+ rows(0).colTail = cols(1)
+ cols(size - 1).colTail = rows(1)
+
+ def get(row: Int, col: Int): Double = {
+ require(row >= 0 && col >= 0)
+ require(row < size && col < size)
+
+ if (col == 0) rows(row).value
+ else if (row == 0) cols(col).value
+ else {
+ val prev = findPreviousOnRow(row, col)
+ if (prev.col == col) prev.value
+ else prev.rowTail match {
+ case null => 0
+ case node => node.value
+ }
+ }
+ }
+
+ def set(row: Int, col: Int, value: Double) {
+ require(row >= 0 && col >= 0)
+ require(row < size && col < size)
+
+ rows(row) synchronized {
+ cols(col) synchronized {
+ if (col == 0) rows(row).value = value
+ else if (row == 0) cols(col).value = value
+ else findWithColNeighbours(row, col) match {
+ /*
+ We should *always* have a left (up) neighbour on the same row (col)
+ (because zeroth rows/columns are treated separately)
+ but the right (down) neighbour might be null or on a different row (col). The left (up)
+ neighbour may have its rowTail (colTail) modified, but we must not mutate the right (down).
+
+ We are synchronized on the current row and column, but the previous node(s) still have
+ free dimensions (e.g. lock on (10,20) but previous row node might be (10, 15) which
+ can also be accessed by lock on (30,15)). This is OK because we're only updating
+ the row tail (in the case of a Node with a free column dimension) or the column tail
+ (in the case of a Node with a free row dimension).
+ */
+ case (up, Some(existing), down) if value == 0 => remove(row, col, up, down)
+ case (up, Some(existing), down) => existing.value = value
+ case (up, None, down) if value == 0 =>
+ case (up, None, down) => insert(row, col, up, down, value)
+ }
+ }
+ }
+ }
+
+ private def remove(row: Int, col: Int, up: Node, down: Node) {
+ findWithRowNeighbours(row, col) match {
+ case (left, Some(_), right) =>
+ up.colTail = down
+ left.rowTail = right
+ case _ => throw new IllegalStateException
+ }
+ }
+
+ private def insert(row: Int, col: Int, up: Node, down: Node, value: Double) {
+ findWithRowNeighbours(row, col) match {
+ case (left, None, right) =>
+ val insert = new Node(row, col, value, right, down)
+ up.colTail = insert
+ left.rowTail = insert
+
+ case _ => throw new IllegalStateException
+ }
+ }
+
+ private def findWithRowNeighbours(row: Int, col: Int): (Node, Option[Node], Node) = {
+ val prev = findPreviousOnRow(row, col)
+ assert(prev.row == row, (row, col) + " but got " + prev.col)
+ if (prev.rowTail == null) (prev, None, null)
+ else if (prev.colTail != null && prev.colTail.row == row && prev.rowTail.col == col) (prev, Some(prev.rowTail), prev.rowTail.rowTail)
+ else (prev, None, prev.rowTail)
+ }
+
+ private def findWithColNeighbours(row: Int, col: Int): (Node, Option[Node], Node) = {
+ val prev = findPreviousOnCol(row, col)
+ assert(prev.col == col, (row, col) + " but got " + prev.col)
+ if (prev.colTail == null) (prev, None, null)
+ else if (prev.rowTail != null && prev.rowTail.col == col && prev.colTail.row == row) (prev, Some(prev.colTail), prev.colTail.colTail)
+ else (prev, None, prev.colTail)
+ }
+
+ private def findPreviousOnRow(row: Int, col: Int): Node = {
+ assert(row > 0 && col > 0)
+ var latest = rows(row)
+ assert(latest.row == row)
+ while (latest.rowTail != null && latest.rowTail.row == row && latest.rowTail.col < col) {
+ latest = latest.rowTail
+ }
+ latest
+ }
+
+ private def findPreviousOnCol(row: Int, col: Int): Node = {
+ assert(row > 0 && col > 0)
+ var latest = cols(col)
+ assert(latest.col == col)
+ while (latest.colTail != null && latest.colTail.col == col && latest.colTail.row < row) {
+ latest = latest.colTail
+ }
+ latest
+ }
+
+}
+
+
+// this is a boolean graph, not a matrix
+class CompressedColumns(val columnSize: Int) extends Logging {
+ self =>
+
+ // gives 139mb of LLists (plus atomic ref array) with 17mb int[] on 2,492,432 refs with 5,819,345 links (201MB heap dump)
+ sealed trait IList {
+ def head: Int
+
+ def tail: IList = ???
+
+ def ::(e: Int): IList = new ILink(e, this)
+
+ def isEmpty: Boolean
+
+ def contains(e: Int): Boolean = false
+
+ def toSet: debox.set.Set[Int] = {
+ debox.set.Set.empty[Int] withEffect { set =>
+ var probe = this
+ while (!probe.isEmpty) {
+ set add probe.head
+ probe = probe.tail
+ }
+ }
+ }
+ }
+
+ object INil extends IList {
+ def head = ???
+
+ def isEmpty = true
+ override def toString = "INil"
+ }
+
+ class ILink(override val head: Int,
+ override val tail: IList) extends IList {
+ def isEmpty = false
+ override def toString = head + "::" + tail
+ }
+
+ private val enters = new AtomicLong
+
+ // don't call until we finish creating!
+ lazy val count: Long = (0L /: cols){_ + _.toSet.length.toLong}
+
+ def entered = enters.get
+
+// private val cols = new AtomicReferenceArray[IList](columnSize)
+ private val cols = Array.fill[IList](columnSize)(INil)
+
+ def get(col: Int) = cols(col).toSet
+
+ def get(row: Int, col: Int): Boolean = get(col)(row)
+
+// private val locks = Array.fill(Runtime.getRuntime.availableProcessors()) {
+// new ReentrantLock()
+// }
+// private def getLock(col: Int) = locks(col % locks.length)
+
+ def set(row: Int, col: Int) {
+ // (0,0) is null
+ require(row > 0)
+ require(col > 0 && col < columnSize, col + ">= " + columnSize)
+
+ // we intentionally allow dupes for insertion speed (e.g. class / array can ref same thing twice)
+// logger.info(row + " " + col)
+ cols.update(col, row :: cols(col))
+// logger.info(row + " appended to " + col + " = " + cols(col))
+
+// if (!cols.compareAndSet(col, null, row :: INil)) {
+ // if (!cols.compareAndSet(col, null, row :: INil)) {
+// val lock = getLock(col)
+// lock.lock()
+// try {
+// val existing = cols.get(col)
+// cols.set(col, row :: existing)
+// } finally lock.unlock()
+// }
+ enters.getAndIncrement()
+ }
+}
View
11 src/main/scala/org/shelmet/heap/shared/ClassType.scala
@@ -6,8 +6,9 @@ object ClassType {
// [[I
// [Lcom.foo.Bar;
// com.foo.Bar
-
- if(sig.startsWith("[")) {
+ if (sig.contains("/"))
+ parse(sig.replace("/", "."))
+ else if (sig.startsWith("[")) {
val subType = sig.drop(1)
val containedTypeChar = subType.charAt(0)
if(containedTypeChar == '[') {
@@ -42,14 +43,14 @@ abstract class ArrayClassType[C] extends ClassType {
def containedType : C
}
-class ObjectArrayClassType(val containedType : ClassType) extends ArrayClassType[ClassType] {
+case class ObjectArrayClassType(containedType : ClassType) extends ArrayClassType[ClassType] {
override def toString = containedType.toString + "[]"
}
-class BaseArrayClassType(val containedType : BaseFieldType) extends ArrayClassType[FieldType] {
+case class BaseArrayClassType(containedType : BaseFieldType) extends ArrayClassType[FieldType] {
override def toString = containedType.typeName + "[]"
}
-class SimpleClassType(classname : String) extends ClassType {
+case class SimpleClassType(classname : String) extends ClassType {
override def toString = classname
}
View
7 src/main/scala/org/shelmet/heap/shared/FieldType.scala
@@ -17,11 +17,12 @@ sealed abstract class FieldType(val jvmTypeChar : Char,val typeName : String) {
// class representing base type fields (e.g int,short,boolean
sealed abstract class BaseFieldType(jvmTypeChar : Char,typeName : String,val fieldSize : Int) extends FieldType(jvmTypeChar,typeName)
-case object BooleanFieldType extends BaseFieldType('Z',"boolean",1)
-case object ByteFieldType extends BaseFieldType('B',"byte",1)
+// these are 64 bit UNIX field sizes
+case object BooleanFieldType extends BaseFieldType('Z',"boolean",4)
+case object ByteFieldType extends BaseFieldType('B',"byte",2)
case object ShortFieldType extends BaseFieldType('S',"short",2)
case object CharFieldType extends BaseFieldType('C',"char",2)
-case object IntFieldType extends BaseFieldType('I',"int",4)
+case object IntFieldType extends BaseFieldType('I',"int",8)
case object LongFieldType extends BaseFieldType('J',"long",8)
case object FloatFieldType extends BaseFieldType('F',"float",4)
case object DoubleFieldType extends BaseFieldType('D',"double",8)
View
74 src/test/scala/org/shelmet/heap/parser/LinkedMatrixTest.scala
@@ -0,0 +1,74 @@
+package org.shelmet.heap.parser
+
+import com.typesafe.scalalogging.slf4j.Logging
+import scala.util.Random._
+import org.scalatest.FunSuite
+
+class LinkedMatrixTest extends FunSuite with Logging {
+
+ test("set and get") {
+ val matrix = new LinkedMatrix(1000)
+ (1 to 10000) foreach { i =>
+ val row = nextInt(1000)
+ val col = nextInt(1000)
+ val value = nextGaussian()
+ matrix.set(row, col, value)
+ assert(matrix.get(row, col) == value, (row, col))
+ }
+ }
+
+ test("parallel set with serial get") {
+ val matrix = new LinkedMatrix(100)
+ val indices = {
+ (1 to 10000) map { i =>
+ (nextInt(100), nextInt(100))
+ }
+ }.toSet
+ val expected = {
+ indices.par map { case (row, col) =>
+ val value = nextGaussian()
+ ((row, col) -> value) withEffect { _ =>
+ matrix.set(row, col, value)
+ }
+ }
+ }.seq
+ expected foreach { case ((row, col), value) =>
+ assert(matrix.get(row, col) == value, (row, col))
+ }
+ }
+}
+
+class CompressedColumnsTest extends FunSuite with Logging {
+
+ test("set and get") {
+ val matrix = new CompressedColumns(0)
+ (1 to 10000) foreach { i =>
+ val row = nextInt(1000)
+ val col = nextInt(1000)
+ matrix.set(row, col)
+ assert(matrix.get(row, col), (row, col))
+ }
+ }
+
+ test("parallel set with serial get") {
+ val matrix = new CompressedColumns(0)
+ val indices = {
+ (1 to 10000) map { i =>
+ (nextInt(100), nextInt(100))
+ }
+ }.toSet
+ indices.par foreach { case (row, col) =>
+ matrix.set(row, col)
+ }
+ indices foreach { case (row, col) =>
+ assert(matrix.get(row, col), (row, col))
+ }
+ for {
+ i <- 0 until 100
+ j <- 0 until 100
+ if !indices.contains(i, j)
+ } {
+ assert(!matrix.get(i, j), (i, j))
+ }
+ }
+}
Something went wrong with that request. Please try again.