v0.4.4

Set version 0.4.4

v0.4.3

Merge with develop v0.4.3

v0.4.2

Annotatio

This is a patch release with few new features and some bug fixes.

• New configurator with some improvements for current and future extensions.
• New parameters in the configuration file for better scheduler setup and control execution.
• Added control interpolation, which allows to call some variables in such control parameters as source_name, preparation_sql, prerequisite_sql.
• Updated some dependencies, including flask_httpauth to fix the issue with authentication.
• Fix missing HTML templates.

Important Changes

Note that this release brings some configuration changes described below.

Configuration

• Added SCHEDULER section with the following parameters: control_parallelism, refresh_interval, maintenance_interval, database_report_interval.

See commits of this release here.

v0.4.1

This is a bug fix release.

v0.4.0

Annotatio

This is a new release with a lot of new features, bug fixes and improvements.

• Case Definition in Analysis Controls. Now controls are more about cases than errors. Users can define a list of cases at the control level and map each data record to a specific case. In turn, each case has its own textual representation shown in the result tables. On top of that, a case can have one of the following types: Normal, Info, Error, Warning, Incident, Discrepancy. Cases other than Normal are automatically saved into the result table as control errors.
• Error Definition now can be done by SQL-like expressions instead of using JSON configuration, which is kept anyway for API or possible Service integrations.
• Datasources can be filtered now with SQL-like expressions inside the control configuration table to avoid extra usage of views instead of ordinary tables.
• Now users can configure a control Preparation SQL which performs before the control. This allows to trigger some things before the control or even execute PL/SQL ETL procedures to prepare control datasources.
• Now users can configure a control Prerequisite SQL to check some conditions before the control. If this check is not passed, then control is not resumed.
• Redesigned the control spawn method, which now not only launches but also handles the control in a separate background threads and, as a result, releases the main thread or user API request.
• Control results now can be deleted each time the control executes.
• Control result tables now can be dropped and re-created each time the control executes.
• Faster deletion of control results that must not be kept more than a day.
• Now PL/SQL HOOK procedure become POSTRUN hook combined with a new PRERUN hook. Thanks to @krokce. The former need_hook parameter from the control configuration table became a global value defining the status of those two.
• Better control configuration audit. Now users can see when and who has updated the control. Also, OS logins now used instead of DB usernames.
• Fixed control cleanup functionality. Thanks to @krokce.
• Fixed database connection overflow and timeout issue when exceptions in threads occur.
• Added new parameters to the configuration file for better database connection setup.
• Extended the max control name length to 45 characters, modified other parameter types and length limits for better user experience and Oracle 12.2+ compatibility.

Important Changes

This release brings some notable database schema and configuration changes described below.

Database

• New fields in RAPO_CONFIG: source_filter, source_filter_a, source_filter_b, case_config, result_config, with_deletion, with_drop, need_prerun_hook, need_postrun_hook, preparation_sql, prerequisite_sql, updated_by, updated_date.
• Some of the VARCHARs in RAPO_CONFIG become CLOBs, some others have increased in length. See the updated DB schema file.
• New fields in RAPO_LOG: prerequisite_value, text_message.
• New dictionary with Case types description RAPO_REF_CASES.
• New triggers to catch RAPO_CONFIG changes.
• Added RAPO_PRERUN_CONTROL_HOOK function. Renamed RAPO_CONTROL_HOOK to RAPO_POSTRUN_CONTROL_HOOK.

Configuration

• Added DATABASE parameters: driver_name, client_path, max_identifier_length, max_overflow, pool_pre_ping, pool_size, pool_recycle, pool_timeout.
• Changed DATABASE parameters: username (former user), vendor_name (vendor), service_name (service).

See commits of this release here.

v0.3.3

• Force multiprocessing to use spawn method in Control

v0.3.2

• Add control cancelation feature
• Fix output table creation

v0.3.1

• Fix username definition in runs without a controlling terminal

v.0.3.0

• Add web API with commands: run-control, get-running-controls, revoke-control-run.
• Add an interface for data read as a Reader class.
• Add a special maintainer thread to the Scheduler.
• Add an automatic cleaner of Control results that can be configured for each control using RAPO_CONFIG.DAYS_RETENTION.
• Add methods for Control results revoke and delete: Control.revoke() and Control.delete(). Revoked results must have X status in RAPO_LOG.
• New Control type REP which stands for "Report" and have only FETCHED metric calculated.
• Now RAPO_CONFIG.MISMATCH_CONFIG is RAPO_CONFIG.RULE_CONFIG.
• Now RAPO_CONFIG.MISMATCH_CONFIG is RAPO_CONFIG.ERROR_CONFIG.
• Now output table names start with a prefix RAPO_REST instead of RAPO_RES in case of single output Control.
• Apply SQL formatting for quieries written in logs.
• Code refactoring of Scheduler and Control interface classes.
• Clean database Oracle schema from weak and unused objects.
• Small fixes and cosmetic improvements.

v.0.2.5

• Added parameters in rapo.ini for usage SQLite as a backend DB.
• Changed some dependencies from 3rd party to native Python.
• Changed scheduler start behaviour.
• Fixed result table creation when no output columns were configured.
• Fixed schema.sql.
• New parameter in rapo.ini for DB service name.
• Small changes in code and docstrings.