Skip to content

Commit

Permalink
ERTool Updates + DataPower (Analytics.xml) Updates
Browse files Browse the repository at this point in the history 
ERTool Updates:
- New Feature: Results now supported mapped tables (table layout of results with a top row using ConditionName's as headers)
- New Feature: Condition element supports attribute TableMapPosition (-2 to ignore it as a mapped table position, 0 or higher to map it to set position as row header)
- New Feature: Max Formula Runtimes can now be set, was previously hardcoded as 15 minutes, now is 5 minutes by default.  Command line argument -formulamaxrunseconds [#] (in seconds).
- When formulas timeout, the resulting report will have a sample result (no conditions declared) and have a URL expressing formula timed out, specifying the section/input file that took too long.
- New Feature: ERTool Analytics support to extract all files of an archive or sections of a document (eg. error report).  Command line argument -retrieveallfiles [true|false].  Note: This is likely to cause a longer period of time to generate results, time will be reflected on the first formula ran with a section wildcard.
- Formula results based on MIME section now gives the section filename visibly instead of saying 'File available here'.. now 'DPMon', 'XACTLog', etc.
- Unnecessary table data in analyticsform.txt and AnalyticsResults.java removed
- Fixed when no command line arguments are being passed to ERTool throwing an ambiguous error, now displays -help.
- Help menu updated to include new arguments, correct spacing and distinguish Analytics arguments.
- build.xml updated for Java 1.8 (prior Java 1.7)
- Stream to String updated to use Future, limit 60 seconds in attempt to get input stream to string.  This avoids a deadlock due to calling IOUtils.toString on some archive files/contents
- Input Stream passed into inputStreamXmlEncapsulate closed after converting stream to string (seems to resolve issues with reading the archive after the deadlock above).

DataPower (Analytics.xml) Updates:
- Tracking of APAR's IT29214 (TAM, other sidecars leak handlers causing interfaces to be missing) and IT29236 (transaction-timeout and persistent connection usage can cause premature closure of transaction)
- Power Supply Events (IPMISelEvents sub-group) added to hardware category
- IPMISelEvents display updated to mapped table layout
- IPMI formulas updated to check for years 2019,2020 (removed 2018)
- FailureNotificationStatus formulas updated to support v2018
- Shutdown reload/reboot/halt formula updated to allow the -f argument and not including a time to shutdown.
- 4 connection failure formulas updated to use mapped table layout.
- GatewayTransactions formula added utilizing mapped table layout.
- XML manager stylesheet cache at full capacity moved from error to critical level.
- ObjectStatus formula tracking RBMSettings updated to remove wildcard (only need default domain) and did a xpath match to RBMSettings to filter out other xpath results.
- ObjectStatus formula tracking 'is down' updated to mapped table format and xpath updated to track a sublist of downed objects from the overall group.
  • Loading branch information
@dmicale
dmicale committed Sep 22, 2019
1 parent ea937a9 commit 9b6eb5d
Show file tree
Hide file tree
Showing 91 changed files with 1,094 additions and 445 deletions.
251 changes: 179 additions & 72 deletions framework/Analytics.xml
View file

Large diffs are not rendered by default.

4 changes: 2 additions & 2 deletions framework/build.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -67,10 +67,10 @@

<target name="compile" depends="init" description="compile source">
<!-- Compile the java code from ${src} into ${build} -->
<javac srcdir="${src}" destdir="${bin}" source="1.7" target="1.7" compiler="javac1.7">
<javac includeantruntime="false" srcdir="${src}" destdir="${bin}" source="1.8" target="1.8" compiler="javac1.8">
<classpath refid="libraries.path" />
</javac>
<javac srcdir="${tools}" destdir="${toolsbin}" source="1.7" target="1.7" compiler="javac1.7">
<javac includeantruntime="false" srcdir="${tools}" destdir="${toolsbin}" source="1.8" target="1.8" compiler="javac1.8">
<classpath refid="libraries.path" />
</javac>
</target>
Expand Down
14 changes: 7 additions & 7 deletions framework/src/Analytics/analyticsform.txt
View file
Original file line number Diff line number Diff line change
@@ -1,13 +1,13 @@
</head><body onload='loadDefault();'>
<form name="selection" id="selection">

<input hidden type="checkbox" class="checkallnone" name="allnone" onChange="if(this.checked) { setAllChecked(1) } else { setAllChecked(0) }" onClick="if(this.checked) { setAllChecked(1) } else { setAllChecked(0) }" value="" checked/></td>
<input hidden type="checkbox" class="checkfield" name="network" onClick="if(this.checked) { updateFields(1, this.value) } else { updateFields(0, this.value) }" value="network" checked/></td>
<input hidden type="checkbox" class="checkfield" name="cpuload" onClick="if(this.checked) { updateFields(1, this.value) } else { updateFields(0, this.value) }" value="cpuload" checked/></td>
<input hidden type="checkbox" class="checkfield" name="memory" onClick="if(this.checked) { updateFields(1, this.value) } else { updateFields(0, this.value) }" value="memory" checked/></td>
<input hidden type="checkbox" class="checkfield" name="hardware" onClick="if(this.checked) { updateFields(1, this.value) } else { updateFields(0, this.value) }" value="hardware" checked/></td>
<input hidden type="checkbox" class="checkfield" name="configuration" onClick="if(this.checked) { updateFields(1, this.value) } else { updateFields(0, this.value) }" value="configuration" checked/></td>
<input hidden type="checkbox" class="checkfield" name="failure" onClick="if(this.checked) { updateFields(1, this.value) } else { updateFields(0, this.value) }" value="failure" checked/></td>
<input hidden type="checkbox" class="checkallnone" name="allnone" onChange="if(this.checked) { setAllChecked(1) } else { setAllChecked(0) }" onClick="if(this.checked) { setAllChecked(1) } else { setAllChecked(0) }" value="" checked/>
<input hidden type="checkbox" class="checkfield" name="network" onClick="if(this.checked) { updateFields(1, this.value) } else { updateFields(0, this.value) }" value="network" checked/>
<input hidden type="checkbox" class="checkfield" name="cpuload" onClick="if(this.checked) { updateFields(1, this.value) } else { updateFields(0, this.value) }" value="cpuload" checked/>
<input hidden type="checkbox" class="checkfield" name="memory" onClick="if(this.checked) { updateFields(1, this.value) } else { updateFields(0, this.value) }" value="memory" checked/>
<input hidden type="checkbox" class="checkfield" name="hardware" onClick="if(this.checked) { updateFields(1, this.value) } else { updateFields(0, this.value) }" value="hardware" checked/>
<input hidden type="checkbox" class="checkfield" name="configuration" onClick="if(this.checked) { updateFields(1, this.value) } else { updateFields(0, this.value) }" value="configuration" checked/>
<input hidden type="checkbox" class="checkfield" name="failure" onClick="if(this.checked) { updateFields(1, this.value) } else { updateFields(0, this.value) }" value="failure" checked/>
<input hidden type="checkbox" class="checkfield" name="general" onClick="if(this.checked) { updateFields(1, this.value) } else { updateFields(0, this.value) }" value="general" checked/>
<input hidden type="checkbox" class="checkfield" name="critical" onClick="if(this.checked) { updateFields(1, this.value) } else { updateFields(0, this.value) }" value="critical" checked/>
<input hidden type="checkbox" class="checkfield" name="error" onClick="if(this.checked) { updateFields(1, this.value) } else { updateFields(0, this.value) }" value="error" checked/>
Expand Down
6 changes: 3 additions & 3 deletions framework/src/Analytics/analyticsjsheader.txt
View file
Original file line number Diff line number Diff line change
Expand Up @@ -439,9 +439,9 @@ if ( mem != null )
var startAt = 0;
for(startAt=0;startAt<test.length;startAt++)
{
if ( test[startAt].includes("MemoryStatus") )
break;
}
if ( test[startAt].includes("MemoryStatus") )
break;
}

if ( test.length > (startAt+2) )
{
Expand Down
70 changes: 37 additions & 33 deletions framework/src/ERMessages.properties
View file
Original file line number Diff line number Diff line change
Expand Up @@ -27,39 +27,43 @@ ERTOOL019I=-generate <ip address>:<port> Create one more Error Report
ERTOOL020I=-user <username> User name for -capture or -generate options.
ERTOOL021I=-password <password> Password for -capture or -generate options.
ERTOOL022I=-format HTML|TEXT|CSV Format for output report
ERTOOL023I=-xsl <format>,<Content ID>,<filename> Style sheet path and file name for
ERTOOL024I= given format and content. Wildcards(*) allowed for format and content.
ERTOOL025I= Separate multiple files with ! like -xsl *,*,x.xsl!HTML,Hardware,y.xsl
ERTOOL026I=-analyticsfile <filename> Local Analytics.xml file used to parse error report
ERTOOL027I=-outfile <outfilename> Destination file for output of analyticsfile processing
ERTOOL028I=-transactions Parses input using dptransx.xml to generate transactions.txt, add -transxml to get an xml format.
ERTOOL029I=-timeformat EST|UTC|MNT|... Time format for transactions output (EST/UTC/MNT/etc.)
ERTOOL030I=-printconditions showall|hidedefault|hideall Generates detailed output for formula debugging
ERTOOL031I=-loglevel debug|info Log level for command line output in analytics formula processing
ERTOOL032I
ERTOOL033I=Defaults:
ERTOOL034I=-help is the default option.
ERTOOL035I=5550 is the default port.
ERTOOL036I=TEXT is the default format used with the -file option.
ERTOOL037I="admin" is default for both -user and -password options.
ERTOOL038I=UTF-8 is the default encoding.
ERTOOL039I=Gzip is the assumed format for report files ending in .gz
ERTOOL040I=Text is the assumed format for report files not ending in .gz
ERTOOL041I=All sections are formatted when the -section option is omitted.
ERTOOL042I=
ERTOOL043I=Examples:
ERTOOL044I=1. Display the command line usage.
ERTOOL045I=ertool -help
ERTOOL046I=2. Copy an existing error report from an appliance.
ERTOOL047I=ertool -capture 9.22.73.157:3478 -user guest -password Guest
ERTOOL048I=3. Generate and copy a new error report from an appliance default port as user "admin".
ERTOOL049I=ertool -generate 11.22.33.44 -password Administrator1
ERTOOL050I=4. Format a saved report as HTML.
ERTOOL051I=ertool -file error-report.txt.gz -format HTML -xsl HTML,*,./src/erHTML.xsl >report.htm
ERTOOL052I=5. Format the Hardware section of a saved report as CSV.
ERTOOL053I=ertool -file error-report.txt.gz -format CSV -section LicenseStatus -xsl CSV,Hardware,./src/erCSV.xsl >Hardware.csv
ERTOOL054I=6. Run Analytics Parser to review error report.
ERTOOL055I=ertool -file error-report.txt.gz -analyticsfile analytics.xml
ERTOOL023I=-xsl <format>,<Content ID>,<filename> Style sheet path and file name for given format and content.
ERTOOL024I= Wildcards(*) allowed for format and content. Separate multiple files with ! like -xsl *,*,x.xsl!HTML,Hardware,y.xsl
ERTOOL025I=
ERTOOL026I=Analytics Options:
ERTOOL027I=-analyticsfile <filename> Local Analytics.xml file used to parse error report
ERTOOL028I=-outfile <outfilename> Destination file for output of analyticsfile processing
ERTOOL029I=-transactions Parses input using dptransx.xml to generate transactions.txt, add -transxml to get an xml format.
ERTOOL030I=-timeformat EST|UTC|MNT|... Time format for transactions output (EST/UTC/MNT/etc.)
ERTOOL031I=-printconditions showall|hidedefault|hideall Generates detailed output for formula debugging
ERTOOL032I=-loglevel debug|info Log level for command line output in analytics formula processing
ERTOOL033I=-gui Presents analytics GUI prompt
ERTOOL034I=-formulamaxrunseconds [#] Set the maximum time in seconds an analytics formula attempts to run, default: 300 seconds
ERTOOL035I=-retrieveallfiles [true|false] In Analytics extract all sections from input data to related_files directory, despite formula match.
ERTOOL036I
ERTOOL037I=Defaults:
ERTOOL038I=-help is the default option.
ERTOOL039I=5550 is the default port.
ERTOOL040I=TEXT is the default format used with the -file option.
ERTOOL041I="admin" is default for both -user and -password options.
ERTOOL042I=UTF-8 is the default encoding.
ERTOOL043I=Gzip is the assumed format for report files ending in .gz
ERTOOL044I=Text is the assumed format for report files not ending in .gz
ERTOOL045I=All sections are formatted when the -section option is omitted.
ERTOOL046I=
ERTOOL047I=Examples:
ERTOOL048I=1. Display the command line usage.
ERTOOL049I=ertool -help
ERTOOL050I=2. Copy an existing error report from an appliance.
ERTOOL051I=ertool -capture 9.22.73.157:3478 -user guest -password Guest
ERTOOL052I=3. Generate and copy a new error report from an appliance default port as user "admin".
ERTOOL053I=ertool -generate 11.22.33.44 -password Administrator1
ERTOOL054I=4. Format a saved report as HTML.
ERTOOL055I=ertool -file error-report.txt.gz -format HTML -xsl HTML,*,./src/erHTML.xsl >report.htm
ERTOOL056I=5. Format the Hardware section of a saved report as CSV.
ERTOOL057I=ertool -file error-report.txt.gz -format CSV -section LicenseStatus -xsl CSV,Hardware,./src/erCSV.xsl >Hardware.csv
ERTOOL058I=6. Run Analytics Parser to review error report.
ERTOOL059I=ertool -file error-report.txt.gz -analyticsfile analytics.xml
ERTOOL001E=Missing file name. Usage: -file <filename>
ERTOOL002E=Unrecognized format {0}. Usage: -format HTML|TEXT|CSV
ERTOOL003E=Firmware {0} not detected. Exiting.
Expand Down
19 changes: 14 additions & 5 deletions framework/src/com/ibm/datapower/er/Analytics/AnalyticsFunctions.java
View file
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
/**
* Copyright 2014-2016 IBM Corp.
* Copyright 2014-2020 IBM Corp.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
Expand Down Expand Up @@ -36,6 +36,7 @@

import com.ibm.datapower.er.ERFramework;
import com.ibm.datapower.er.Analytics.ConditionsNode.LogLevelType;
import com.ibm.datapower.er.Analytics.MappedCondition.MAPPED_TABLE_POSITION;
import com.ibm.datapower.er.Analytics.Structure.Formula;
import com.ibm.datapower.er.Analytics.Structure.RunFormula;

Expand Down Expand Up @@ -386,9 +387,17 @@ public static ConditionField getConditionField(Node condNode) {
// conversion operation
String conversionType = getAttributeByName(condNode, "Conversion").toLowerCase();

int defaultMapPosition = MAPPED_TABLE_POSITION.AUTO_SET.getType();
// conversion operation
String mappedPosition = getAttributeByName(condNode, "TableMapPosition").toLowerCase();
try {
defaultMapPosition = Integer.parseInt(mappedPosition);
} catch (Exception ex) {
}

ConditionField field = new ConditionField(pos, conditionFieldPosition, regGroupValue, conditionName,
conditionOperation, conditionValue, conditionRegEXP, conditionNextOperation, conversionType,
inValueSetting);
inValueSetting, defaultMapPosition);

return field;
}
Expand Down Expand Up @@ -541,9 +550,9 @@ public static void setupNodeVariables(ERFramework framework, ConditionsNode node
if (framework.GetID() > 0) {
reportFile = "ReportFile" + framework.GetID();
}
node.addCondition("SectionName", sectionName);
node.addCondition("ReportFile", reportFile);
node.addCondition("ReportFileName", fileName);
node.addCondition("SectionName", sectionName, MAPPED_TABLE_POSITION.NO_SET.getType());
node.addCondition("ReportFile", reportFile, MAPPED_TABLE_POSITION.NO_SET.getType());
node.addCondition("ReportFileName", fileName, MAPPED_TABLE_POSITION.NO_SET.getType());
}

public static void setupNodeVariables(RunFormula formula, ConditionsNode node, String sectionName) {
Expand Down
Loading

1 comment on commit 9b6eb5d

@dmicale
Copy link
Contributor Author

@dmicale dmicale commented on 9b6eb5d Sep 22, 2019 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As a comment the mapped table formatting for 'text' output is only tabbed and doesn't really give a true table format. There will be a solution in a future ERTool Update as HTML formatting was priority given time constraints and is the primarily use-case for ERTool Analytics right now. #11 opened on this concern.

Please sign in to comment.