openHistorian v2.5

@ritchiecarroll ritchiecarroll released this Jun 1, 2018 · 57 commits to master since this release

Official stable release of version 2.5 of the openHistorian (v2.5.6).

NOTE: if you are upgrading from a prior version of the openHistorian older than 2.4, you must migrate your existing configuration to use the latest database schema when the Configuration Setup Utility is run after the installation completes - this version includes schema updates.

This version of the openHistorian allows Grafana to be seamlessly integrated such that the openHistorian primary web site can act as a reverse proxy to an instance of Grafana accessible from the main web URL, e.g.: http://localhost:8180/grafana/. When configured in this mode, the openHistorian will auto-launch the grafana-server executable and act as a front-end server for Grafana. Additionally, the openHistorian will maintain user security synchronization such that a user with an Administrator role in openHistorian will also have an Admin role in Grafana, or if a user has an Editor role in openHistorian they will have an Editor role in Grafana, and so on.

See the new YouTube video that walks users through the process of installing and configuring the openHistorian and covers the process of building custom displays from openHistorian data using Grafana visualizations.

Deployments of the openHistorian with hosted Grafana integration include pre-configured data sources for the primary data and statistics archives named OHDATA and OHSTAT respectively. The latest versions of the openHistorian and OSIsoft-PI Grafana data source are included, as well as a default System Status dashboard:

systemstatus

New Features and Improvements

  • Improved operations when storing archive data on remote Linux drives
  • Improved SignalR API for trend and export web screens by reducing user based caching
  • Improved configuration setup utility when migrating databases using local users in SQL Server
  • Improved web based security including better application of CORS and anti-forgery tokens within self-hosted web architecture
  • Updated external dependencies, both assemblies and Javascript libraries, to latest versions for improved security
  • Improved hosted Grafana user security synchronization
  • Improved included system statistics Grafana dashboard to display better total active devices
  • Release built with version 2.2 of GSF, see full release notes

Other New Features / Fixes (available since 2.4)

  • Improved web based authentication
  • Fully overhauled system security
  • Modbus device connections
  • COMTRADE data file importer
  • Improved sub-second time handling
  • Included GSF time-series Filter Adapter management
  • Added panning support for Trend Measurements screen
  • Added new Grafana functions and capabilities (see full list)
  • Better pre-compilation options of Razor web pages at service start
  • Provided CSV export options to daily quality reporting
  • Added "useSourcePrefixNames" boolean connection string parameter to DataSubscriber to allow override ability on subscription prefixes applied to devices names
  • Included new GSF adapter to allow selection of a "best" value based on configuration options
  • Updated POSIX based installations to default to Basic authentication
  • Modified the latest values cache in DynamicCalculator to be fed by the concentration engine to resolve usability issues
  • Updated Grafana annotation queries to always operate at full resolution so that no alarms will be missed
  • Fixed memory leak in MultiProtocolFrameParser when replay start time and stop time is specified
  • Fixed logic error in AverageFrequency calculator when reporting unreasonable results as NaN
  • Fixed SQL syntax for updated Phasor synchronization in DataSubscriber
  • Updated LDAP security provider to allow UserData.IsDefined to be assumed to be true if no cached user can be found to allow authentication to be attempted
  • Added a safety check around target cache acquisition for cases where cache may have been disposed
  • Handled edge case in SIDToAccountName to resolve errors with null input
  • Removed references and dependencies on to Thread.CurrentPrincipal throughout system as part of security overhaul
  • Corrected access control issues on shared Modbus configuration screen

Migrating to openHistorian 2.0 in other GPA Products

It is recommended to install a stand-alone instance of the openHistorian 2.0 and subscribe to other data sources, e.g., using GEP, to get data in the openHistorian. However, using the openHistorian.Binaries.zip download that can be found here, you can extract the needed openHistorian 2.0 assemblies needed to upgrade any openHistorian 1.0 instance running in a GPA product built using the GSF Time-Series Library, e.g., the common PPA openHistorian 1.0 instance running in an openPDC. You can also migrate existing openHistorian 1.0 .D data files into openHistorian 2.0 .D2 data files using the Archive Upgrade Utility that comes installed with the openHistorian.

To start using the openHistorian 2.0 within an existing GSF Time-Series Library based application, begin by following these steps.

openHistorian v2.4

@ritchiecarroll ritchiecarroll released this Jan 1, 2018 · 277 commits to master since this release

Official stable release of version 2.4 of the openHistorian (v2.4.30).

NOTE: if you are upgrading from a prior version of the openHistorian, you must migrate your existing configuration to use the latest database schema when the Configuration Setup Utility is run after the installation completes - this version includes schema updates.

This version of the openHistorian allows Grafana to be seamlessly integrated such that the openHistorian primary web site can act as a reverse proxy to an instance of Grafana accessible from the main web URL, e.g.: http://localhost:8180/grafana/. When configured in this mode, the openHistorian will auto-launch the grafana-server executable and act as a front-end server for Grafana. Additionally, the openHistorian will maintain user security synchronization such that a user with an Administrator role in openHistorian will also have an Admin role in Grafana, or if a user has an Editor role in openHistorian they will have an Editor role in Grafana, and so on.

Deployments of the openHistorian with hosted Grafana integration include pre-configured data sources for the primary data and statistics archives named OHDATA and OHSTAT respectively. The latest versions of the openHistorian and OSIsoft-PI Grafana data source are included, as well as a default System Status dashboard

New Features and Improvements

  • Improved web based authentication
  • Fully overhauled system security
  • Modbus device connections
  • COMTRADE data file importer
  • Improved sub-second time handling
  • Included GSF time-series Filter Adapter management
  • Added panning support for Trend Measurements screen
  • Added new Grafana functions and capabilities (see full list)
  • Better pre-compilation options of Razor web pages at service start
  • Provided CSV export options to daily quality reporting
  • Added "useSourcePrefixNames" boolean connection string parameter to DataSubscriber to allow override ability on subscription prefixes applied to devices names
  • Included new GSF adapter to allow selection of a "best" value based on configuration options

Important Fixes

  • Updated POSIX based installations to default to Basic authentication
  • Modified the latest values cache in DynamicCalculator to be fed by the concentration engine to resolve usability issues
  • Updated Grafana annotation queries to always operate at full resolution so that no alarms will be missed
  • Fixed memory leak in MultiProtocolFrameParser when replay start time and stop time is specified
  • Fixed logic error in AverageFrequency calculator when reporting unreasonable results as NaN
  • Fixed SQL syntax for updated Phasor synchronization in DataSubscriber
  • Updated LDAP security provider to allow UserData.IsDefined to be assumed to be true if no cached user can be found to allow authentication to be attempted
  • Added a safety check around target cache acquisition for cases where cache may have been disposed
  • Handled edge case in SIDToAccountName to resolve errors with null input
  • Removed references and dependencies on to Thread.CurrentPrincipal throughout system as part of security overhaul
  • Corrected access control issues on shared Modbus configuration screen

Migrating to openHistorian 2.0 in other GPA Products

It is recommended to install a stand-alone instance of the openHistorian 2.0 and subscribe to other data sources, e.g., using GEP, to get data in the openHistorian. However, using the openHistorian.Binaries.zip download that can be found here, you can extract the needed openHistorian 2.0 assemblies needed to upgrade any openHistorian 1.0 instance running in a GPA product built using the GSF Time-Series Library, e.g., the common PPA openHistorian 1.0 instance running in an openPDC. You can also migrate existing openHistorian 1.0 .D data files into openHistorian 2.0 .D2 data files using the Archive Upgrade Utility that comes installed with the openHistorian.

To start using the openHistorian 2.0 within an existing GSF Time-Series Library based application, begin by following these steps.

Known Issues

The built-in System Status dashboard "Active Input Connections" single-stat panel reports an invalid value for system with multiple direct connections or GEP based inputs, change the metric query to the following to produce a correct count:

SliceCount(10, Exclude(0, 0, FILTER ActiveMeasurements WHERE PointTag LIKE '%!PMU:ST4'))

To change the query you may need to first make the page “editable”. To do so, click the “gear” icon at the top of the page and select “Make Editable”, save the dashboard, then refresh the page. Now when you click on the panel titled “Active Input Connections” you can click “Edit” and paste in the corrected metric query above.


This release has been updated, note the following changes since v2.4.22:

  • Fixed references for TrendMeasurements view to correct zoom out and pan
  • Updated Grafana deployment to 4.6.3
  • Reduced web file deployment footprint

openHistorian v2.3

@chefsteph9 chefsteph9 released this Jul 12, 2017 · 676 commits to master since this release

Official stable release of version 2.3 of the openHistorian (v2.3.7).

See full GSF change log since last release.

New Features and Improvements

  • Simplified Datahub record operations with new modeled table functions.
  • Updated to Owin 3.1.0 and added Cors assembly to allow for cross-site scripting.
  • Added security mode indication, e.g., FIPS compliance, to primary web interface home page.
  • Added optional timestamp reasonability checking for incoming measurements to be archived.
  • Made status messages more resistant to possible exceptions.
  • Suppressed blank warnings and messages from flowing to the console.

Important Fixes

  • Start the Web UI on a separate thread when starting the service.
  • Fixed issue where html was using forbidden characters for ids.
  • Added more error handling to the Configuration Setup Utility SqlServer setup screen.
  • Configuration Setup Utility updates or creates AssemblyBindings node in .config file.
  • Fixed the status lights on the Graph Measurements page of the Web UI.
  • Add missing "System.Web.Cors.dll" to the installer.
  • Fixed TrenValueAPI static constructor.
  • Used FilePath.GetFiles() in ArchiveList2.AttachFileOrPath() so that permissions issues encountered during recursive search don't cause the openHistorian to fail to attach accessible archive files.
  • Updated SqlClr script to include GSF.Core.SqlClr.dll and added assembly to installation package.
  • Fixed issues when system is set to use FIPS security.
  • Corrected TableOperation models based on latest attribute namespace changes.
  • In the CSU, fixed database setup screens to retain the value of the Encrypted option on the connection string when using an existing database as-is.
  • Added invokable 'CurtailArchiveFiles' command so that daily file curtailment can be ran immediately.
  • Fixed spelling of 'EnableTimeReasonabilityCheck' setting.
  • Fixed errors in CSU when reading encrypted configuration file settings.

Migrating to openHistorian 2.0 in other GPA Products

Using the openHistorian.Binaries.zip download that can be found below, you can extract the needed openHistorian 2.0 assemblies needed to upgrade any openHistorian 1.0 instance running in a GPA product built using the GSF Time-Series Library, e.g., the common PPA openHistorian 1.0 instance running in an openPDC. You can also migrate existing openHistorian 1.0 .D data files into openHistorian 2.0 .D2 data files using the Archive Upgrade Utility that comes installed with the openHistorian.

To start using the openHistorian 2.0 within an existing GSF Time-Series Library based application, begin by following these steps.

openHistorian v2.2

@ritchiecarroll ritchiecarroll released this Mar 17, 2017 · 820 commits to master since this release

Official stable release of version 2.2 of the openHistorian (v2.2.25).

See full change log since last release.

New Features and Improvements

  • GrafanaAdapters: Added full set of aggregation and operational functions that operate on a per-series or per-set for all data sources that implement Grafana interfaces. See documentation.
  • LogFileViewer: Added new enhanced general logging capabilities along with a new application, Log File Viewer, that is used to analyze logs.
  • Web UI: Updated home page to include time displays in both UTC and local time.
  • GSF.TimeSeries: Added up-time statistics for system, subscriber, publisher, input streams and output streams.
  • GSF.TimeSeries: Added a new enhanced version of the time-series special compression algorithm, automatically enabled for new connections, used with the Gateway Exchange Protocol that uses much less bandwidth. Add ; compression=true to connection string of existing GEP subscriptions to enable.
  • GSF.TimeSeries: Created a lookup cache for the main adapter meta-data using dictionary based lookups. This improved load times for input adapters on large systems (e.g., from 160 seconds to less than 1 second in some cases).
  • GSF.TimeSeries: Updated measurement definitions to carry meta-data as part of measurement key with an internal reference inside measurement implementations to reduce total memory footprint requirements for new measurements.
  • GSF.Core: Added a static timer class to allow many hundreds of scheduled events that run at the same interval to be associated with a single wait event instead of creating a new timer for each event (e.g., as hundreds of devices needing to check statistics every second) - this alleviates thread pool exhaustion that can occur when hundreds of timer events get executed at the same time.

Important Fixes

  • GSF.SortedTreeStore: Fixed bug in UnionTreeStreamSortHelper (found by @yuwenpeng) that could cause occasional data loss when duplicate records were being written to the openHistorian - in cases of encountered duplicates (i.e., by key of time and ID), there was a possibility that the duplicate record could cause a rollover operation to end prematurely.
  • DataSubscriber: Updated measurements received with GEP to apply any linear adjustments defined in the local configuration.
  • GSF.Core: Added code to ensure that log files won't become deadlocked if an error occurs during rollover.
  • GSF.TimeSeries: Updated DataSubscriber meta-data synchronization to exclude calculation and alarm signals from the device measurement removal processing such that if a local subscription creates new calculations associated with a subscribed device, the locally created measurements will not get automatically removed at the next synchronization.

Migrating to openHistorian 2.0 in other GPA Products

Using the openHistorian.Binaries.zip download that can be found below, you can extract the needed openHistorian 2.0 assemblies needed to upgrade any openHistorian 1.0 instance running in a GPA product built using the GSF Time-Series Library, e.g., the common PPA openHistorian 1.0 instance running in an openPDC. You can also migrate existing openHistorian 1.0 .D data files into openHistorian 2.0 .D2 data files using the Archive Upgrade Utility that comes installed with the openHistorian.

To start using the openHistorian 2.0 within an existing GSF Time-Series Library based application, begin by following these steps.

openHistorian v2.1

@ritchiecarroll ritchiecarroll released this Oct 16, 2016 · 1125 commits to master since this release

Note: This is not the latest version.

Official stable release of version 2.1 of the openHistorian (v2.1.1).

See full change log since last release.

New Features and Improvements

  • Includes new self-hosted web architecture which enables functionality for web based UIs and APIs, such as Data Trending and Export screen.
  • New web interfaces to support Grafana dashboards
  • Support for PostgreSQL configuration databases
  • Bulk power calculation adapter improvements including a more functional AdjustmentStrategy parameter for line-to-line and line-to-neutral adjustments
  • New dynamic e-mail responder adapter that will send an e-mail based on a custom expression
  • Simplified installation deployments signed with extended validation certificate

Important Fixes

  • Improved performance of secure implementations of Gateway Exchange Protocol for access control list lookups in service and manager implementations
  • Better phasor reduction management in manager, output streams and GEP meta-data synchronization (e.g., reducing output stream to only include positive sequence phasors)
  • Fixed issues in openHistorian 1.0 data and statistic archive deployments with potential race conditions between rollovers and reading data to prevent data access issues
  • Outage logs now properly serialize stop timestamps in UTC ensuring proper data gap recovery operations
  • NT SERVICE accounts are now assigned needed rights in installer to restart associated host service allowing restart buttons to properly function
  • Fixed issues related to creating and using a custom abstract synchrophasor configurations

Remote Console Access for Users with Viewer Roles

A new command has been added to the remote console to allow messages to be filtered down to desired results, see Filter command syntax below. In order for users with view role access to access the remote console, the filter command will need to be added to the included security resources in the main configuration file.

The simplest way to add this command is to run the XML Configuration Editor application that can be found on the start menu and add the Filter command before the Settings command in the IncludedResources configuration key in the securityProvider section of the configuration settings:

<securityProvider>
    <add name="IncludedResources" value="UpdateSettings,UpdateConfigFile=Special; Filter,Settings,Schedules,Help,Status,Version,Time,User,Health,List,Invoke,ListCommands=*; Processes,Start,ReloadCryptoCache,ReloadSettings,ResetHealthMonitor,Connect,Disconnect,Initialize,ReloadConfig,Authenticate,RefreshRoutes,TemporalSupport,LogEvent=Administrator,Editor; *=Administrator"
        description="Semicolon delimited list of resources to be secured along with role names." encrypted="false" />
</securityProvider>

Filter Command Syntax

Filters status messages coming from the service.

   Usage:
       Filter [ { -List |
                  -Include <FilterDefinition> |
                  -Exclude <FilterDefinition> |
                  -Remove <ID> } ... ]

       Filter -?

       FilterDefinition ::= Type { Alarm | Warning | Information } |
                            Message <FilterSpec> |
                            Regex <FilterSpec>

   Options:
       -List        Displays a list of the client's active filters
       -Include     Defines a filter matching messages to be displayed
       -Exclude     Defines a filter matching messages to be suppressed
       -Remove      Removes a filter

openHistorian v2.0.392

@ritchiecarroll ritchiecarroll released this Sep 5, 2016 · 1255 commits to master since this release

Note: This is not the latest version.

This release of the openHistorian that includes a self-hosted web based UI*. Although the new web UI is pre-release, it is functional with a fast data export. The openHistorian engine is very stable and this new version has had all the improvements included with updates to the Grid Solutions Framework libraries, including:
https://github.com/GridProtectionAlliance/gsf/wiki/Changes-since-v2.1

If you are upgrading from an older version we highly recommend migrating your database to a new schema during the configuration setup process:

  • Select "I want to use an existing configuration"
  • Choose "Database" option
  • Select "I want to upgrade to the latest schema"

The new UI has screens for trending and exporting data:

Select Measurements

Trend Measurements

Export Measurements

* The new web based UI is designed to run with a newer HTML 5 capable browser.