Support for Filter Groups (#4 and #10) and connection specific settin…

…gs (#8)
OraOpenSource · Oct 8, 2015 · 72966bf · 72966bf
1 parent 609cb09
commit 72966bf
Show file tree

Hide file tree

Showing 4 changed files with 185 additions and 73 deletions.
diff --git a/README.md b/README.md
@@ -3,7 +3,7 @@ The purpose of this project is to create a export of an APEX application in JSON
 
 _This project is still undergoing active development. As such, configuration and command line options may change._
 
-# Prerequistes
+# Prerequisites
 
 ## SQLcl
 This projects requires that [SQLcl](http://www.oracle.com/technetwork/developer-tools/sql-developer/downloads/index.html) (_Sep 23, 2015 or above_) is installed. It is used for its ability to quickly output queries in JSON format and cursor support.
@@ -13,53 +13,75 @@ Mac users can find additional information on how to install and configre SQLcl [
 To find the current version of SQLcl, simply run SQLcl and it will be displayed. Minimum required version is `SQLcl: Release 4.2.0.15.265.1501 RC`.
 
 ## Node.js
-Node.js version 0.12.x or greater is required. To find your current version run:
+[Node.js](https://nodejs.org) version 0.12.x or greater is required. To find your current version run:
 
 ```bash
 node --version
 ```
 
 # Running
-They're two ways to create a JSON output of an APEX appliaction. SQLcl can be used to call the `apex-diff.sql` file or Node.js can be used. It is recommended that you use the Node.js application as it handles some additional tasks (such as prettifying the JSON file) behind the scene.
 
-_Note: This project may eventually be listed on npm for easy install._
+## Download
+Either download this project from GitHub or clone it using git:
+`git clone https://github.com/OraOpenSource/apex-diff.git`
 
-## Node.js App
-A [Node.js](https://nodejs.org) application has been included with this project and is the recommended method to run APEX Diff.
+_Note: This project may eventually be listed on npm for easy install._
 
-### Config
+## Config
 Create a new (or copy `config_sample.json`) config file in the `config` folder called `config.json`.
 
 - `debug`: optional, boolean, default `false`.
   - If `true` will output each step of the process.
-- `rebuildTempFile` : optional, boolean, default `false`.
-  - If `true`, the temp sql file will be re-generated. Unless upgrdaing APEX, it is not recommended to set this to `true` as it takes additional time to generate the temp sql file.
+- `rebuildTempFile` : optional, boolean, default `true`.
+  - If `true`, the temp sql file will be re-generated. Unless upgrading APEX, it is not recommended to set this to `true` as it takes additional time to generate the temp sql file.
 - `sqlcl` : optional, string, default `sql`.
   - Command name (or full path to) SQLcl file.
-- `connections` : required, JSON object.
-  - Name/value pair for each database connection.
-- `filters` : options, array of regular expressions to remove objects from JSON file
-  - The filter will be applied on both the `apex_view_name` and the `apex_view_name.column_name`
-  - It is case insensitive
-  - In the example below, the filter will remove all columns that contain `updated` in it (this is `updated_by` and `updated_on`) as well as the `apex_application_lists` view.
+- `filters` : optional, array of regular expressions to remove objects from JSON file
+  - The filter will be applied on both the `apex_view_name` and the `apex_view_name.column_name`.
+  - Filters will be applied for all connections.
+  - It is case insensitive.
   - Any `\` needs to be escaped with `\\` as the regular expression must also be a valid JSON string.
+- `filterGroups` : optional, JSON object, defining set of filterGroups.
+  - Define a set of filters that can then be easily applied to specific connections
+  - Each entry is a name/value pair.
+  - Value is an array of filters that correspond to the filterGroup
+- `connections` : required, JSON object.
+  - Name/value pair for each database connection or connection object
+  - Connection Object: Use this for specific connection information for a connection
+    - `connectionDetails`: required, database connection
+    - `filters`: optional, array of filters (see `filters` above)
+    - `filterGroups`: optional, array of list of `filterGroups` to apply
+
 
 Example:
 ```json
 {
   "sqlcl" : "sqlcl",
   "connections" : {
-    "dev" : "giffy/giffy@localhost:11521/xe",
+    "dev" : {
+      "connectionDetails" : "giffy/giffy@localhost:11521/xe",
+      "filters" : ["apex_application_pages"],
+      "filterGroups" : ["updateInfo"]
+    },
     "prod" : "oos/oos@prod.oraopensource.com:1521/xe"
   },
-  "filters" : [
-    ".*\\.updated.*",
-    "^apex_application_lists$"
-  ]
+  "filters" : ["apex_application_lists"],
+  "filterGroups" : {
+    "updateInfo" : [".+\\..+_updated_.*"]
+  }
 }
 ```
 
-### Run
+Example Explanation:
+
+- `filters`: For all connections data for `apex_application_lists` will be removed.
+- `connections`: They're two connections, `dev` and `prod`,
+  - `dev`:
+    - `filters`: Just for this connection, all entries for `apex_application_pages` will be removed.
+    - `filterGroups`: Any filters defined in the `filterGroup` `updateInfo` will be applied. In this example, all column names that contain `_updated_` will be removed.
+- `filterGroups`: A filterGroup called `updateInfo` has been created and can be applied to individual connections.
+
+## Run
 The Node.js application requires two parameters:
 
 - `connection name`: This is the name that is found in the `connections` object in `config.json`.
@@ -74,21 +96,23 @@ node ~/Documents/GitHub/apex-diff/app.js dev 113
 
 This will generate a prettified JSON file: `f113.json`.
 
-## SQLcl
-To run the application directly using SQLcl:
+# Developers
+To help with future development, the following configuration can be added to `config.json`:
 
-```sql
-<sqlcl command> <connection string> @<path to apex-diff.sql> <app_id> <temp spool filename>
-```
+- `dev`
+  - `runSql`: `boolean` - If false, will skip over the SQL command. This is useful for testing JSON parsing.
+  - `saveJson`: `boolean` - If false, will not save any changes to the JSON file.
 
-Example: _note: I renamed the sql file to sqlcl._
-```sql
-sqlcl giffy/giffy@localhost:11521/xe @source/apex-diff 113 temp.sql
+Example:
+```json
+...
+"dev" : {
+  "runSql" : false,
+  "saveJson" : true
+}
+...
 ```
 
-This will create a new file, `temp.sql` and `f113.json`. `temp.sql` can be deleted and `f113.json` is the unprettified JSON output of the APEX application. It is up to you to prettify it (or use the Node.js app, above).
-
-
 # Known Issues
 
 ## ORA-00600 Error

diff --git a/app.js b/app.js
@@ -44,6 +44,23 @@ function logTime(){
   debug('Time:', (new Date() - timer.start)/1000 + 's');
 }
 
+// TODO mdsouza:
+//http://stackoverflow.com/questions/1584370/how-to-merge-two-arrays-in-javascript-and-de-duplicate-items
+// Array.prototype.unique = function() {
+//     var a = this.concat();
+//     for(var i=0; i<a.length; ++i) {
+//         for(var j=i+1; j<a.length; ++j) {
+//             if(a[i] === a[j])
+//                 a.splice(j--, 1);
+//         }
+//     }
+//
+//     return a;
+// };
+
+
+// TODO mdsouza: update docs on filter groups
+// TODO mdsouza: include some sample filter groups in config_sample.json
 var
   //Console
   log = console.log,
@@ -82,10 +99,18 @@ var
     configFile : path.resolve(__dirname , 'config/config.json'),
     sqlcl : "sql", //Path to sqlcl
     debug : false,
-    rebuildTempFile : false, // Rebuild the APEX json file
+    rebuildTempFile : true, // Rebuild the APEX json file
     filters : [], // array of filters
-    filtersRegExp : [] // array of filters to Regular Expressions
-  }
+    filterGroups : {},
+    filtersRegExp : [], // array of filters to Regular Expressions
+    connectionDetails : "",
+    // TODO mdsouza: document this for developers
+    dev : {
+      runSql : true,
+      saveJson : true
+    }
+  },
+  temp = ''
 ;
 
 //Additional settings
@@ -98,47 +123,101 @@ if (!fs.existsSync(options.configFile)) {
   console.log('See TODO (doc link) for more info');
   process.exit(1);
 }
-else{
-  // Load config file
-  var configFileData = fs.readFileSync(options.configFile, 'utf8');
-  options = extend(options, JSON.parse(configFileData));
 
-// TODO mdsouza: support for connection specific filters
 
-  // Convert filters to RegExp objects
-  for (var i in options.filters) {
-    options.filtersRegExp[options.filtersRegExp.length] = new RegExp(options.filters[i], 'i');
+// Load config file
+var configFileData = fs.readFileSync(options.configFile, 'utf8');
+options = extend(options, JSON.parse(configFileData));
+
+debug('options extended by confiFile:\n', options, '\n');
+
+debug('\n* Initial Validations*');
+if (!arguments.appId) {
+  console.log('Missing appId');
+  process.exit(1);
+}
+else if (!options.connections[arguments.connectionName]) {
+  console.log('Invalid connection:', arguments.connectionName);
+  console.log('Valid connections:', Object.keys(options.connections).toString());
+  process.exit(1);
+}
+
+
+//Parse connection information
+var curConn = options.connections[arguments.connectionName];
+debug('\ncurConn:', curConn);
+
+if (curConn.constructor !== Object){
+  debug('\ncurConn is string');
+  //Connection info is a string (just connection details)
+  options.connectionDetails = curConn;
+}
+else {
+  debug('\ncurConn is JSON, parsing');
+  //Connection info is a JSON object
+  curConn = extend(
+    {
+      connectionDetails : "",
+      filters : ""
+    },
+    curConn);
+
+  //Current connection is
+  options.connectionDetails = curConn.connectionDetails;
+
+  //Only add filters if exists
+  if (curConn.filters) {
+    options.filters = options.filters.concat(curConn.filters);
   }
 
-  debug('options', options);
+  // Add filters from filterGroups to curConn
+  if (curConn.filterGroups){
+    for (var i in curConn.filterGroups){
+
+      temp = options.filterGroups[curConn.filterGroups[i]];
+      if (temp){
+        options.filters = options.filters.concat(temp);
+      }
+
+    }//for
+  }// filter groups defined
+}//connection info is JSON object
+
+//Make filters unique
+// TODO mdsouza: Make filters unique
+
+// Convert filters to RegExp objects
+debug('\nConverting filters to RegExp\n')
+for (var i in options.filters) {
+  debug('Filter RegExp:', options.filters[i]);
+  options.filtersRegExp[options.filtersRegExp.length] = new RegExp(options.filters[i], 'i');
 }
 
+debug('\noptions', options);
+
 
 //General debug (can do here since value is set)
-debug('sql:', sql);
-debug('arguments:',  arguments);
+debug('\nsql:', sql);
+debug('\narguments:',  arguments);
 
 
 // VALIDATIONS
+debug('\n* Validations *');
+
 // Check that argument has two parameters (connection and appid)
-if (!arguments.connectionName){
-  console.log('Missing connection string');
-  process.exit(1);
-}
-else if (!arguments.appId) {
-  console.log('Missing appId');
-  process.exit(1);
-}
-else if (!options.connections[arguments.connectionName]) {
-  console.log('Invalid connection:', arguments.connectionName);
-  console.log('Valid connections:', Object.keys(options.connections).toString());
+if (!options.connectionDetails){
+  console.log('Missing connection details');
   process.exit(1);
 }
 
-debug('Validations passed');
+debug('\nValidations passed');
 logTime();
 
 
+// // TODO mdsouza:
+// process.exit(1);
+
+
 debug('Setting up sql.command');
 
 // Determine if file should be rebuilt
@@ -156,31 +235,29 @@ else{
 
 //Finish sql.command replacement
 sql.command = sql.command.replace('%SQLCL%', options.sqlcl);
-sql.command = sql.command.replace('%CONNECTION%', options.connections[arguments.connectionName]);
+sql.command = sql.command.replace('%CONNECTION%', options.connectionDetails);
 sql.command = sql.command.replace('%APP_ID%', arguments.appId);
 sql.command = sql.command.replace('%SPOOL_FILENAME%', sql.files.generateJson.fullPath);
 debug('sql.command:', sql.command);
 
-debug('Running sql.command');
-childProcess = execSync(sql.command,{ encoding: 'utf8' });
-debug(childProcess);
-logTime();
 
-debug('Reading .json file');
+if (options.dev.runSql){
+  debug('Running sql.command');
+  childProcess = execSync(sql.command,{ encoding: 'utf8' });
+  debug(childProcess);
+  logTime();
+}
+
+debug('\nReading .json file');
 sql.files.json.data = fs.readFileSync(sql.files.json.fileName, 'utf8');
 //Convert string to JSON
 sql.files.json.data = JSON.parse(sql.files.json.data);
 logTime();
 
 
-// TODO mdsouza: add option that connections can be a straight connection or an JSON object that contains connectionDetails, filters
-
-
-debug('Filtering JSON');
+debug('\nFiltering JSON');
 for (var apexView in sql.files.json.data.items[0]){
   // This loop is the list of APEX views.
-  // TODO mdsouza: remove re
-
   // If delete slows things down can look at setting to undefined
   // http://stackoverflow.com/questions/208105/how-to-remove-a-property-from-a-javascript-object
   //Check to see about removing entire APEX view itself.
@@ -207,8 +284,10 @@ logTime();
 sql.files.json.data = (JSON.stringify(sql.files.json.data, null, 2));
 logTime();
 
-debug('Writing pretty JSON to file');
-fs.writeFileSync(sql.files.json.fileName, sql.files.json.data);
+if(options.dev.saveJson){
+  debug('Writing pretty JSON to file');
+  fs.writeFileSync(sql.files.json.fileName, sql.files.json.data);
+}
 
 // END
 logTime();
diff --git a/config/config_sample.json b/config/config_sample.json
@@ -1,7 +1,14 @@
 {
   "connections" : {
     "sample" : "user/pass@server:port/sid",
-    "dev" : "giffy/giffy@dev.oraopensource.com:11521/xe",
+    "dev" : {
+      "connectionDetails" : "giffy/giffy@dev.oraopensource.com:11521/xe",
+      "filters" : ["apex_application_pages"],
+      "filterGroups" : ["updated"]
+    },
     "prod" : "giffy/giffy@prod.oraopensource.com:11522/xe"
+  },
+  "filterGroups" : {
+    "updateInfo" : [".+\\..+_updated_.*"]
   }
 }
diff --git a/source/apex-diff.sql b/source/apex-diff.sql
@@ -158,4 +158,6 @@ spool off
 
 @&SPOOL_FILENAME. &APP_ID.
 
+-- TODO mdsouza: add option to export APEX application as well (.sql file)
+
 exit