MONDRIAN

   Added ObjectFactory and Java5 memory monitoring support.
   Now when using Java5, if a query is using "too much" memory
   a MemoryLimitExceededException is thrown rather than
   an OutOfMemoryError.

[git-p4: depot-paths = "//open/mondrian/": change = 8645]

doc/configuration.html

@@ -38,6 +38,10 @@ <h3>Contents</h3>
     <li><a href="#Cache_management">Cache_management</a><ol>
 			<li><a href="#Schema_cache">Schema_cache</a></li>
     	</ol>
+    	</ol>
+    <li><a href="#Memory_management">Memory_management</a><ol>
+			<li><a href="#Out_of_memory">Out_of_memory</a></li>
+    	</ol>
     </li>
 </ol>
 <!--
@@ -277,14 +281,6 @@ <h3>1.1 Property list<a name="Property_list">&nbsp;</a></h3>
         and will be treated as a null member if they are later referenced in
         a query.</td>
     </tr>
-    <tr>
-      <td><a href="api/mondrian/olap/MondrianProperties.html#IterationLimit">
-      <code>mondrian.rolap.IterationLimit</code></a></td>
-      <td>int</td>
-      <td>0</td>
-      <td>If > 0, the maximum number of iterations allowed when evaluating an
-        aggregate.  If 0, there is no limit.</td>
-    </tr>
     <tr>
       <td colspan="4">
 
@@ -724,24 +720,48 @@ <h3>1.1 Property list<a name="Property_list">&nbsp;</a></h3>
       <td>1,000</td>
       <td>Limit on the number of rows returned by XML/A drill through request.</td>
     </tr>
+    <tr>
+      <td style="vertical-align: top;" colspan="4">
+
+      <b><br>
+      Memory monitoring</b>
+      </td>
+  </tr>
+    <tr>
+      <td><code>
+      <a href="api/mondrian/olap/MondrianProperties.html#MemoryMonitor">
+      mondrian.util.memoryMonitor.enable
+      </a></code></td>
+
+      <td>boolean</td>
+      <td>true</td>
+      <td>If enabled and one is using Java5 or above, then Mondrian
+      will use the Java5 memory monitoring capability - a
+      MemoryLimitExceededException exception ought be throw rather than
+      a OutOfMemoryError when memory get low and the JVM will not
+      crash.
+      </td>
+    </tr>
+    <tr>
+      <td><code><a href="api/mondrian/olap/MondrianProperties.html#MemoryMonitorThreshold">
+      mondrian.util.memoryMonitor.percentage.threshold
+      </a></code></td>
+
+      <td>int</td>
+      <td>90</td>
+      <td>At what level of memory usage should Mondrian be notified that
+      memory is running low.</td>
+    </tr>
+
 </table>
 
 <h2>Connect strings<a name="Connect_strings">&nbsp;</a></h2>
 
 <h3>Connect string syntax<a name="Connect_string_syntax">&nbsp;</a></h3>
 
 <p>Mondrian connect strings are a connection of property/value pairs, of the 
-form 'property=value;property=value;...'. For example:</p>
-
-<blockquote><code>Provider=Mondrian;
-    Jdbc='jdbc:mysql://localhost/foodmart';
-    JdbcUser=foodmart;
-    JdbcPassword=foodmart;
-    JdbcDrivers=com.mysql.jdbc.Driver;
-    Catalog=file:demo/FoodMart.xml</code>
-</blockquote>
-
-<p>Values can be enclosed in single-quotes, which allows them to contain spaces
+form 'property=value;property=value;...'.</p>
+<p>Values can be enclosed in single-quotes, which allows them to contain spaces 
 and punctuation. See the the
 <a target="_blank" href="http://msdn.microsoft.com/library/en-us/oledb/htm/oledbconnectionstringsyntax.asp">
 OLE DB connect string syntax specification</a>.</p>
@@ -934,6 +954,48 @@ <h3>Schema cache<a name="Schema_cache"></a></h3>
 
 </font>
 
+<h2>Memory management<a name="Memory_management"></a></h2>
+
+<h3>Out Of Memory<a name="Out_of_memory"></a></h3>
+
+<p>Java <code>OutOfMemoryError</code>s have always been an issue with
+applications. When the JVM throws an <code>Error</code> as 
+oppose to an <code>Exception</code> it is telling the application
+that its world has ended and it has no recourse but to die.
+Prior to Java5 there was not much one could do other
+than buy 64-bit machines with lots of RAM and hope for the best.
+For a multi-user, Mondrian environment with potentially very large 
+data-sets and clients that
+can generate queries requesting arbitrarily large amounts of that
+data, this can be an issue. This is especially the case when
+Mondrian is being hosted on some corporate web-server; applications
+that kill web-servers are not looked upon favorably by IT.
+<p>
+With Java5 (and Java6, etc.) there is alternative. An application 
+cay take advantage of
+a new feature in Java5 allowing the application to be notified when
+memory starts running low. This allows the application to take
+preemptive action prior to an <code>OutOfMemoryError</code> being
+generated by the Java runtime. 
+<p>
+Mondrian takes advantage of this new feature. Rather than
+passing an <code>OutOfMemoryError</code> to its client, it
+will now stop processing the present query, free up data structures
+associated with the present query and return a
+<code>MemoryLimitExceededException</code> to the client.
+The <code>MemoryLimitExceededException</code> is one of Mondrian's
+<code>ResultLimitExceededException</code> which are used to communicate
+with clients that a limit has been exceeded, in this case, memory
+usage.
+<p>
+By default, for Mondrian running under Java5, this feature is
+enabled and the "safety limit" is set at 90 percent, when 
+memory usage gets to with 90 percent of the maximum possible, the
+the processing of the current query is stopped and a
+<code>MemoryLimitExceededException</code> is return to the client.
+See the Memory monitoring properties above on this page 
+for additional information.
+
 <hr noshade size="1"/>
 <p>
     Author: Julian Hyde; last modified August 2006.<br/>

doc/developer_notes.html

@@ -27,6 +27,7 @@ <h2>Contents</h2>
   <li><a href="#Log_Levels">Logging Levels and Information</a></li>
   <li><a href="#Agg_default_rules">Default aggregate table recognition rules</a></li>
   <li><a href="#Snowflakes">Snowflakes and the DimensionUsage level attribute</a></li>
+  <li><a href="#Memory_monitoring">Memory Monitoring</a></li>
 </ol>
 
 <h2>Logging Levels and Information<a name="Log_Levels">&nbsp;</a></h2>
@@ -963,6 +964,32 @@ <h2>Snowflakes and the DimensionUsage level attribute<a name="Snowflakes">&nbsp;
 element's table attribute MUST use the table alias and NOT the table name. 
 </p>
 
+<h2>Memory monitoring, Java5 and memory usage<a name="Memory_monitoring">&nbsp;</a></h2>
+<p>
+With Java5, developers using its memory monitoring capabilities 
+need to make sure the code they create will best use this new feature.
+In particular, if a given algorithm which uses significant memory
+is surrounded by 
+block in which a <code>MemoryMonitor.Listener</code> has been
+registered with the <code>MemoryMonitor</code>, then the code
+must periodically check if a memory notification has occurred.
+If the algorithm has long stretches of allocating memory for
+data structures that will exist throughout the life-time of the
+algorithm's execution during which it does not check for 
+memory notifications, then it is possible that an 
+<code>OutOfMemoryError</code> could still occur.
+You can see for the ResultSet object where, basically, all memory
+is created in its constructor, throughout the Member determination
+and value evaluation code, the Query object's checkCancelOrTimeout
+method is called repeatedly.
+<p>
+The Java5 memory management mechanism is not fool proof, so to speak.
+If one, as an example, attempts to allocate a very big
+array, an <code>OutOfMemoryError</code> will occur. This
+technique works best when memory is allocated incrementally between
+checks for memory notifications allowing the developer to 
+take steps before a possible OOME gotterdammerung.
+
 <hr noshade size="1"/>
 <p>
     Author: Julian Hyde, Richard Emberson; last updated August, 2006.<br/>

src/main/mondrian/olap/ConnectionBase.java

@@ -27,11 +27,16 @@
  */
 public abstract class ConnectionBase implements Connection {
 
+    public static void memoryUsageNotification(Query query, String msg) {
+        query.setOutOfMemory(msg);
+    }
+
     protected ConnectionBase() {
     }
 
     protected abstract Logger getLogger();
 
+
     public String getFullConnectString() {
         String s = getConnectString();
         String catalogName = getCatalogName();

src/main/mondrian/olap/MemoryLimitExceededException.java

@@ -0,0 +1,30 @@
+/*
+// $Id$
+// This software is subject to the terms of the Common Public License
+// Agreement, available at the following URL:
+// http://www.opensource.org/licenses/cpl.html.
+// Copyright (C) 2004-2005 TONBELLER AG
+// All Rights Reserved.
+// You must accept the terms of that agreement to use this software.
+*/
+package mondrian.olap;
+
+/**
+ * Exception which indicates some resource limit was exceeded.
+ * When a client receives a <code>MemoryLimitExceededException</code> the state
+ * of the objects associated with the query execution can NOT be
+ * counted on being correct - specifically data structures could be
+ * in an inconsistent state or missing entirely. No attempt should be
+ * make to access or use the result objects.
+ *
+ * @version $Id$
+ */
+public class MemoryLimitExceededException 
+        extends ResultLimitExceededException {
+
+    public MemoryLimitExceededException(String message) {
+        super(message);
+    }
+}
+
+// End MemoryLimitExceededException.java

src/main/mondrian/olap/MondrianProperties.java

@@ -824,6 +824,19 @@ public Property getPropertyDefinition(String path) {
      */
     public final IntegerProperty IterationLimit = new IntegerProperty(
         this, "mondrian.rolap.iterationLimit", 0);
+
+    /**
+     * Whether the <code>MemoryMonitor</code> should be enabled. By
+     * default for Java5 and above it is enabled.
+     */
+    public final BooleanProperty MemoryMonitor = new BooleanProperty(
+        this, "mondrian.util.memoryMonitor.enable", true);
+
+    /**
+     * The default <code>MemoryMonitor</code> percentage threshold.
+     */
+    public final IntegerProperty MemoryMonitorThreshold = new IntegerProperty(
+        this, "mondrian.util.memoryMonitor.percentage.threshold", 90);
 }
 
 // End MondrianProperties.java

src/main/mondrian/olap/Query.java

@@ -111,6 +111,12 @@ public class Query extends QueryPart {
      */
     private boolean isCanceled;
 
+    /**
+     * If true, this query was notified that it might cause an 
+     * OutOfMemoryError.
+     */
+    private String outOfMemoryMsg;
+
     /**
      * If true, query is in the middle of execution
      */
@@ -292,6 +298,10 @@ public void cancel() {
         isCanceled = true;
     }
 
+    void setOutOfMemory(String msg) {
+        outOfMemoryMsg = msg;
+    }
+
     /**
      * Checks if either a cancel request has been issued on the query or
      * the execution time has exceeded the timeout value (if one has been
@@ -314,6 +324,9 @@ public void checkCancelOrTimeout() {
                     (long) queryTimeout / 1000);
             }
         }
+        if (outOfMemoryMsg != null) {
+            throw new MemoryLimitExceededException(outOfMemoryMsg);
+        }
     }
 
     /**

src/main/mondrian/rolap/RolapConnection.java

@@ -13,6 +13,8 @@
 package mondrian.rolap;
 
 import mondrian.olap.*;
+import mondrian.util.MemoryMonitor;
+import mondrian.util.MemoryMonitorFactory;
 import org.apache.commons.dbcp.ConnectionFactory;
 import org.apache.commons.dbcp.DataSourceConnectionFactory;
 import org.apache.commons.dbcp.DriverManagerConnectionFactory;
@@ -369,7 +371,31 @@ public Object getProperty(String name) {
      * the property file
      */
     public Result execute(Query query) {
+        class Listener implements MemoryMonitor.Listener {
+            private final Query query;
+            Listener(final Query query) {
+                this.query = query;
+            }
+            public void memoryUsageNotification(long used, long max) {
+                StringBuffer buf = new StringBuffer(200);
+                buf.append("OutOfMemory used=");
+                buf.append(used);
+                buf.append(", max=");
+                buf.append(max);
+                buf.append(" for connection: ");
+                buf.append(getConnectString());
+                // Call ConnectionBase method which has access to
+                // Query methods.
+                RolapConnection.memoryUsageNotification(query, buf.toString());
+            }
+        }
+        Listener listener = new Listener(query);
+        MemoryMonitor mm = MemoryMonitorFactory.instance().getObject();
         try {
+            mm.addListener(listener);
+            // Check to see if we must punt
+            query.checkCancelOrTimeout();
+
             if (LOGGER.isDebugEnabled()) {
                 LOGGER.debug(Util.unparse(query));
             }
@@ -391,6 +417,9 @@ public Result execute(Query query) {
             query.setQueryEndExecution();
             return result;
 
+        } catch (ResultLimitExceededException e) {
+            // query has been punted
+            throw e;
         } catch (Exception e) {
             String queryString;
             query.setQueryEndExecution();
@@ -401,6 +430,8 @@ public Result execute(Query query) {
             }
             throw Util.newError(e, "Error while executing query [" +
                     queryString + "]");
+        } finally {
+            mm.removeListener(listener);
         }
     }