This repository contains ArcGIS geoprocessing tools and scripts to execute Esri Data Reviewer from Workflow Manager on a Roads and Highways ALRS. These validation tools are implemented using an ArcGIS Python Toolbox in ArcGIS Desktop 10.5.1. Esri does a nice job documenting Python Toolboxes.
The WMXDataReviewerTools repository is the source code for the NYSDOT Validations Toolbox, which executes various Data Reviewer jobs against the Milepoint LRS. The validations and the code within this repository make strong assumptions regarding the schema of the data, thus this toolbox is not expected to "just run" on other datasets. However, it should serve as a fine example of some methods used to employ Data Reviewer in a Workflow Manager Workflow.
The toolbox consists of four tools:
This tool selects the edits made by a specific user since a specific date (using the EDITED_BY and EDITED_DATE columns on LRSN_Milepoint), buffers the edits by 10 meters, and executes a specified Reviewer Batch Job file against all features that intersect the buffer polygons. This geoprocessing workflow means that the RBJ will validate routes additional to the edits directly conducted in the current WMX job. View the table below for a detailed list of validations that are in the current RBJ.
In the above figure, the yellow route was edited by the current user in the current WMX job. The blue colored polygon represents the 10 meter buffer that is created while executing the Execute Reviewer Batch Job on R&H Tool from this toolbox. When this polygon is passed into the Execute Reviewer Batch Job geoprocessing tool, it will validate the yellow route plus all of the red routes in this diagram. The black routes will not be validated.
This tool selects the edits made by a specific user since a specific date (using the EDITED_BY and EDITED_DATE columns on LRSN_Milepoint) and runs them through a set of Python functions that validate specific data relationships. Examples include "Local Roads should not have a ROUTE_NUMBER", "COUNTY_ORDER should be a series of numbers incrementing by a value of one on a DOT_ID", the "ROUTE_SUFFIX of a ROADWAY_TYPE=Route must be None", and many other validations. View the source code or the table below for a complete list.
These validations are executed against the entire LRSN_Milepoint table, validating only active routes. There are two SQL queries that are executed as apart of this validation tool, one of which ensures there is only one combination of roadway-level attributes, the other of which ensures there is not more than one combination of DOT_ID, COUNTY_ORDER, and DIRECTION.
This tool serves as a wrapper function for items 1 through 3 of these lists. The data flow of this tool requires the tools to be executed in the following order:
- Execute Roadway Level Attribute Validations
- Execute Reviewer Batch Job on R&H Edits
- Execute SQL Validations Against Network.
This toolbox is mainly designed to run within a Workflow Manager Workflow, which is called when a Workflow Manager Job is created by the user. The typical workflow will create a unique database version for the user, create a corresponding Reviewer Workspace, launch ArcMap with the proper version and workspace so that the user can conduct their edits, launch this Python Toolbox as a geoprocessing tool, launch ArcMap again so the user can see the results of the validations, and finally close the version and the Workflow Manager job.
In NYSDOT's current development workflow, the Execute All Validations tool is called. You can see the schematics of the workflow below:
In NYSDOT's current Temporary Production workflow, Execute Reviewer Batch Job on R&H edits is called. You can see the schematics of the workflow below:
These tools can be used on an ad-hoc basis from within ArcGIS Desktop or ArcCatalog. It's important to understand the input parameters for the tool prior to execution, as they're designed to be pre-populated by Workflow Manager. Higher level documentation can be viewed in the tool help, but here's a brief description:
| Parameter | Description |
|---|---|
| job__started_date | The Milepoint feature class will be filtered with EDITED_DATE >= job__started_date |
| job__owned_by | The Milepoint feature class will be filtered with EDITED_BY = job__owned_by |
| job__id | The job__id will be used to construct the SDE version name for the edits. It assumes names like "SVC{job__owned_by.upper()}".HDS_GENERAL_EDITING_JOB_{job__id}, where .upper() indicates all capital letters and job__id will correspond with the Workflow Manager Job ID. The Job ID is also used by Data Reviewer when naming a Reviewer Session that is created in the workflow. The DR Session will typically have a name like Session 23 : 23232. |
| production_ws | SDE file pointing to the R&H ELRS geodatabase |
| production_ws_version | A string. Value must be one of ELRS.Lockroot or Edit Version (Determined from WMX Job ID). It is suggested to use ELRS.Lockroot for better Data Reviewer exception management. The database version name is stored in the reviewer workspace for each record, thus running all validations on Lockroot helps to ensure Data Reviewer will recognize duplicates. If you validate the edit version, the Version name will be in the Reviewer record. Since the version name will always be different, DR will not recognize the records as duplicates. |
| reviewer_ws | SDE file or file geodatabase for the Data Reviewer results |
| log_path | If a log file is desired, it must have a .txt extension |
| log_level | DEBUG or INFO - This controls the Geoprocessing Tool's logging to the Arc Dialog and the output log file |
| batch_job_file | Path to a file with .rbj extension, which is a Reviewer Batch Job created in ArcMap |
| full_db_flag | If True, the filtering described in the first two rows is disregarded and all active features are validated |
As an example, let's say the user SVC\jdoe has created a new Workflow Manager General Editing Job and conducted some Roads and Highways edits within the new version. The job was created on 2/29/2020 and assigned the system generated ID 23232. J. Doe would like to validate those edits by running Execute All Validations. They would populate the tool's UI as follows:
A Reviewer Batch Job is created using ArcGIS Desktop. Esri provides pre-programmed routines that search for hard-to-see errors in your data. Common examples are "dangles" (i.e. polylines that almost connect and probably were intended to connect) and "cutbacks" (i.e. the digitizer accidentally put a z shape into the polyline vertices).
The Reviewer Batch Job portion of the WMXDataReviewer Tools is the most malleable/easily-extensible part of the validation workflow. Any user can go into ArcMap, load the correct data, and generate an RBJ to their specifications. The RBJ can then be fed into the tools via Workflow Manager or ad-hoc execution. Additionally, these tools are not required to run RBJ validations. However, Data Reviewer's Workflow Manager integration provides limited options for which features are input into an RBJ. Since running the RBJ on the entire LRS network is time consuming and overwhelms the user with results, this toolbox helps by selecting only the user's edits. When the edits are buffered, the buffer polygon can be fed into the Data Reviewer Execute Reviewer Batch Job GP tool, which will select all intersecting features.
RBJ files are text files in an XML format. These files are most easily read using the Data Reviewer Extension for ArcGIS. Specifically, you'll want to use the Reviewer Batch Job Manager tool from the Data Reviewer toolbar. Learn more about RBJs in the Esri docs
Here is a list of the checks that are currently included in the RBJ:
| Check Title | DR Check Type | Where Clause | Additional Parameters |
|---|---|---|---|
| Multipart Feature | Multipart Feature | ||
| Duplicate Vertices | Duplicate Vertices | Tolerance: 0.0011 meters | |
| Overlapping Centerlines - Overlaps | Overlapping Centerlines | FC1: Centerlines FC2: Centerlines Type: Overlap Attributes: None |
|
| Overlapping Centerlines - Contains | Overlapping Centerlines | FC1: Centerlines FC2: Centerlines Type: Contains Attributes: None |
|
| Invalid Geometry | Invalid Geometry |
| Check Title | DR Check Type | Where Clause | Additional Parameters |
|---|---|---|---|
| Invalid FROM_DATE | Execute SQL | (FROM_DATE IS NULL) OR (FROM_DATE < '01/01/2007') OR (FROM_DATE > TO_DATE) OR (FROM_DATE > CURRENT_TIMESTAMP) | |
| Invalid TO_DATE | Execute SQL | (TO_DATE < '01/01/2007') OR (TO_DATE < FROM_DATE) OR (TO_DATE > CURRENT_TIMESTAMP) | |
| Invalid MEASURE | Execute SQL | (MEASURE IS NULL) OR (MEASURE < 0) OR (MEASURE >= 100) | |
| Invalid Geometry | Invalid Geometry |
| Check Title | DR Check Type | Where Clause | Additional Parameters |
|---|---|---|---|
| Invalid Geometry | Invalid Geometry | (FROM_DATE IS NULL OR FROM_DATE <= CURRENT_TIMESTAMP) AND (TO_DATE IS NULL OR TO_DATE >= CURRENT_TIMESTAMP) | |
| Non-Linear Segments | Non-Linear Segment | (FROM_DATE IS NULL OR FROM_DATE <= CURRENT_TIMESTAMP) AND (TO_DATE IS NULL OR TO_DATE >= CURRENT_TIMESTAMP) | |
| Self-Intersection Check | Polyline or Path Closes on Self Check | (FROM_DATE IS NULL OR FROM_DATE <= CURRENT_TIMESTAMP) AND (TO_DATE IS NULL OR TO_DATE >= CURRENT_TIMESTAMP) | |
| Domain Check | Domain Check | (FROM_DATE IS NULL OR FROM_DATE <= CURRENT_TIMESTAMP) AND (TO_DATE IS NULL OR TO_DATE >= CURRENT_TIMESTAMP) | |
| Duplicate Vertices | Duplicate Vertices | (FROM_DATE IS NULL OR FROM_DATE <= CURRENT_TIMESTAMP) AND (TO_DATE IS NULL OR TO_DATE >= CURRENT_TIMESTAMP) | Tolerance: 0.0011 meters |
| Calibration Point on Routes | Geometry on Geometry | (FROM_DATE IS NULL OR FROM_DATE <= CURRENT_TIMESTAMP) AND (TO_DATE IS NULL OR TO_DATE >= CURRENT_TIMESTAMP) | FC1: Calibration Points FC2: Milepoint Tolerance: 0.0011 meters Compare Attributes: Milepoint.ROUTE_ID=Calibration_Point.ROUTE_ID |
| Cutbacks | Cutbacks Check | (FROM_DATE IS NULL OR FROM_DATE <= CURRENT_TIMESTAMP) AND (TO_DATE IS NULL OR TO_DATE >= CURRENT_TIMESTAMP) | Min angle in degrees: 30 |
| Length Check | Evaluate Polyline Length Check | (FROM_DATE IS NULL OR FROM_DATE <= CURRENT_TIMESTAMP) AND (TO_DATE IS NULL OR TO_DATE >= CURRENT_TIMESTAMP) | Type: Polyline Length: <=0.001 miles |
| Dangles | Find Dangles | (FROM_DATE IS NULL OR FROM_DATE <= CURRENT_TIMESTAMP) AND (TO_DATE IS NULL OR TO_DATE >= CURRENT_TIMESTAMP) | Dangle Tolerance: 0.009 miles |
| Monotinicity Check | Monotinicity Check | (FROM_DATE IS NULL OR FROM_DATE <= CURRENT_TIMESTAMP) AND (TO_DATE IS NULL OR TO_DATE >= CURRENT_TIMESTAMP) | Evaluate: M values Search Goal: Non-monotonic features/Decreasing Values |
| Orphan Check | Orphan Check | (FROM_DATE IS NULL OR FROM_DATE <= CURRENT_TIMESTAMP) AND (TO_DATE IS NULL OR TO_DATE >= CURRENT_TIMESTAMP) | |
| Invalid FROM_DATE | Execute SQL | (FROM_DATE IS NULL) OR (FROM_DATE < '01/01/2007') OR (FROM_DATE > TO_DATE) OR (FROM_DATE > CURRENT_TIMESTAMP) | |
| Invalid TO_DATE | Execute SQL | (TO_DATE < '01/01/2007') OR (TO_DATE < FROM_DATE) OR (TO_DATE > CURRENT_TIMESTAMP) | |
| Invalid FROM_MEASURE | Execute SQL | ROUND(SHAPE.STStartPoint().M, 3) <> 0.000 OR ISNUMERIC(SHAPE.STStartPoint().M) <> 1 | |
| Invalid MEASURE_RANGE | Execute SQL | SHAPE.STStartPoint().M > SHAPE.STEndPoint().M | |
| Invalid TO_MEASURE | Execute SQL | SHAPE.STEndPoint().M IS NULL OR ISNUMERIC(SHAPE.STEndPoint().M) <> 1 | |
| Too Many Vertices | Execute SQL | SHAPE.STNumPoints() > 1500 | |
| Local Road Overlap - Overlaps | Geometry on Geometry | ROADWAY_TYPE = 1 AND ((FROM_DATE IS NULL) OR (FROM_DATE < '01/01/2007') OR (FROM_DATE > TO_DATE) OR (FROM_DATE > CURRENT_TIMESTAMP)) | where_clause: Same on both FCs FC1: Milepoint FC2: Milepoint Spatial Relation Check type: Overlaps Compare Attributes: None |
| Local Road Overlap - Contains | Geometry on Geometry | ROADWAY_TYPE = 1 AND ((FROM_DATE IS NULL) OR (FROM_DATE < '01/01/2007') OR (FROM_DATE > TO_DATE) OR (FROM_DATE > CURRENT_TIMESTAMP)) | where_clause: Same on both FCs FC1: Milepoint FC2: Milepoint Spatial Relation Check type: Contains Compare Attributes: None |
| Check Title | DR Check Type | Where Clause | Additional Parameters |
|---|---|---|---|
| Functional Class | Event Check | (FROM_DATE IS NULL OR FROM_DATE <= CURRENT_TIMESTAMP) AND (TO_DATE IS NULL OR TO_DATE >= CURRENT_TIMESTAMP) |
where_clause: On event and Milepoint Search Goals: Find orphans/Find overlaps/Find Gaps Measure tolerance: 0.000000621369949494949 |
| Municipality | Event Check | (FROM_DATE IS NULL OR FROM_DATE <= CURRENT_TIMESTAMP) AND (TO_DATE IS NULL OR TO_DATE >= CURRENT_TIMESTAMP) |
where_clause: On event and Milepoint Search Goals: Find orphans/Find overlaps/Find Gaps Measure tolerance: 0.000000621369949494949 |
| Maintenance Jurisdiction | Event Check | (FROM_DATE IS NULL OR FROM_DATE <= CURRENT_TIMESTAMP) AND (TO_DATE IS NULL OR TO_DATE >= CURRENT_TIMESTAMP) |
where_clause: On event and Milepoint Search Goals: Find orphans/Find overlaps/Find Gaps Measure tolerance: 0.000000621369949494949 |
| Owning Jurisdiction | Event Check | (FROM_DATE IS NULL OR FROM_DATE <= CURRENT_TIMESTAMP) AND (TO_DATE IS NULL OR TO_DATE >= CURRENT_TIMESTAMP) |
where_clause: On event and Milepoint Search Goals: Find orphans/Find overlaps/Find Gaps Measure tolerance: 0.000000621369949494949 |
| Number of Through Lanes Reverse | Event Check | ((FROM_DATE IS NULL) OR (FROM_DATE < '01/01/2007') OR (FROM_DATE > TO_DATE) OR (FROM_DATE > CURRENT_TIMESTAMP)) AND (DIRECTION = '0' OR DIRECTION = '2') | where_clause: On event and Milepoint Search Goals: Find orphans/Find overlaps/Find Gaps Measure tolerance: 0.000000621369949494949 |
| Number of Through Lanes Primary | Event Check | ((FROM_DATE IS NULL) OR (FROM_DATE < '01/01/2007') OR (FROM_DATE > TO_DATE) OR (FROM_DATE > CURRENT_TIMESTAMP)) AND (DIRECTION = '0' OR DIRECTION = '1') | where_clause: On event and Milepoint Search Goals: Find orphans/Find overlaps/Find Gaps Measure tolerance: 0.000000621369949494949 |
| Width of Through Lanes Reverse | Event Check | ((FROM_DATE IS NULL) OR (FROM_DATE < '01/01/2007') OR (FROM_DATE > TO_DATE) OR (FROM_DATE > CURRENT_TIMESTAMP)) AND (DIRECTION = '0' OR DIRECTION = '2') | where_clause: On event and Milepoint Search Goals: Find orphans/Find overlaps/Find Gaps Measure tolerance: 0.000000621369949494949 |
| Width of Through Lanes Primary | Event Check | ((FROM_DATE IS NULL) OR (FROM_DATE < '01/01/2007') OR (FROM_DATE > TO_DATE) OR (FROM_DATE > CURRENT_TIMESTAMP)) AND (DIRECTION = '0' OR DIRECTION = '1') | where_clause: On event and Milepoint Search Goals: Find orphans/Find overlaps/Find Gaps Measure tolerance: 0.000000621369949494949 |
Roads and Highways provides dialog boxes for users to input roadway attributes. Many of these attributes are partially limited by the use of Esri Attribute Domains. Domains do a good job of making sure the values lie within a specific set, however, they do not limit attributes based on the values of other attributes. Many of the fields in the LRSN_Milepoint table only apply to signed routes, which are indicated by ROADWAY_TYPE=Route. Other fields are assumed to be null when the ROADWAY_TYPE=Road. Further, some fields like ROUTE_ID and DOT_ID allow for freehand input, saving the input as string values to the database. These fields should be numeric in nature, so any alphabetical characters are invalid. Domains cannot account for this.
The Roadway Level Attribute checks are managed in a Python function. The relevant features are read from the LRSN_Milepoint table using arcpy.da.SearchCursor, and the attributes and ROADWAY_TYPE are passed into the function. The function checks for the following relationships:
| Validation | Roadway Type |
|---|---|
| ROUTE_ID must be a nine digit number | Road, Ramp, Route, or Non-Mainline |
| DOT_ID must be a six digit number | Road, Ramp, Route, or Non-Mainline |
| COUNTY_ORDER must be 2 digit zero padded | Road, Ramp, Route, or Non-Mainline |
| COUNTY_ORDER must be greater than 00 | Road, Ramp, Route, or Non-Mainline |
| COUNTY_ORDER should be less than 29 | Road, Ramp, Route, or Non-Mainline |
| COUNTY_ORDER must increment by a value of 1 for this DOT_ID | Road, Ramp, Route, or Non-Mainline |
| COUNTY_ORDER must equal '01' for singular DOT_ID | Road, Ramp, Route, or Non-Mainline |
| COUNTY_ORDER has too many ROUTE_IDs for this DOT_ID | Road, Ramp, Route, or Non-Mainline |
| SIGNING must be null | Road or Ramp |
| ROUTE_NUMBER must be null | Road or Ramp |
| ROUTE_SUFFIX must be null | Road or Ramp |
| ROUTE_QUALIFIER must be 'No Qualifier' | Road or Ramp |
| PARKWAY_FLAG must be No | Road or Ramp |
| ROADWAY_FEATURE must be null | Road or Ramp |
| ROUTE_NUMBER must not be null | Route |
| ROADWAY_FEATURE must be null | Route |
| ROUTE_NUMBER must be '900' route when SIGNING is null | Route |
| SIGNING must be null | Non-Mainline |
| ROUTE_NUMBER must be null | Non-Mainline |
| ROUTE_SUFFIX must be null | Non-Mainline |
| ROUTE_QUALIFIER must be null | Non-Mainline |
| PARKWAY_FLAG must be 'No' | Non-Mainline |
| ROADWAY_FEATURE must not be null | Non-Mainline |
The SQL Validations check the full LRSN_Milepoint table for some essential relationships. While these validations are partially redundant, they provide the added benefit of always validating the entire active route network. This is beneficial, as systems downstream of the R&H interface rely on sound data quality.
These validations are always run against a "Versioned View" of the data. The Esri SDE schema provides this database view so that you can execute SQL against different trees of the data. The version that the versioned view references is set with a stored database procedure. For example, you can see the Lockroot version in the versioned view by executing this SQL: EXEC ELRS.sde.set_current_version "ELRS.Lockroot";.
The first SQL query ensures that there is only one combination of roadway level attributes. This is essential since the LRSN is split at county borders. Thus, downstream routes can easily fall out of sync. It examines the attributes SIGNING, ROUTE_NUMBER, ROUTE_SUFFIX, ROADWAY_TYPE, ROUTE_QUALIFIER, ROADWAY_FEATURE, and PARKWAY_FLAG, while paying no mind to the DOT_ID, ROUTE_ID, and COUNTY_ORDER:
SELECT DOT_ID, COUNT (DISTINCT CONCAT(SIGNING, ROUTE_NUMBER, ROUTE_SUFFIX, ROADWAY_TYPE, ROUTE_QUALIFIER, ROADWAY_FEATURE, PARKWAY_FLAG))
FROM ELRS.elrs.LRSN_Milepoint_evw
WHERE (FROM_DATE IS NULL OR FROM_DATE <= CURRENT_TIMESTAMP) AND (TO_DATE IS NULL OR TO_DATE >= CURRENT_TIMESTAMP)
GROUP BY DOT_ID
HAVING COUNT (DISTINCT CONCAT(SIGNING, ROUTE_NUMBER, ROUTE_SUFFIX, ROADWAY_TYPE, ROUTE_QUALIFIER, ROADWAY_FEATURE, PARKWAY_FLAG))>1;The second query ensures that there is only one combination of DOT_ID, COUNTY_ORDER, DIRECTION. The data model dictates that each DOT_ID corresponds with a roadway. For example, 100495 corresponds to I-87. The data model further dictates that each county border should see the roadways linear representation split, and, if the roadway continues into the next county, the COUNTY_ORDER should increment by a value of 1. Each combination of DOT_ID and COUNTY_ORDER could have up to 2 values for DIRECTION, which should correspond to both the Inventory (primary) and Non-Inventory (reverse) side of a dual carriageway highway. To test these assumptions are not exceeded, we use this query:
SELECT DOT_ID, COUNTY_ORDER, DIRECTION, COUNT (1)
FROM ELRS.elrs.LRSN_Milepoint_evw
WHERE (FROM_DATE IS NULL OR FROM_DATE <= CURRENT_TIMESTAMP) AND (TO_DATE IS NULL OR TO_DATE >= CURRENT_TIMESTAMP)
GROUP BY DOT_ID, COUNTY_ORDER, DIRECTION
HAVING COUNT (1)>1This tool is intended to run via Workflow Manager (WMX). The easiest way to achieve this is to create a new WMX Step Type using the Workflow Manager Administrator. The Step Type can be one of Launch Geoprocessing Tool or Execute Geoprocessing Tool. The Launch GP Tool step will present the user with the GP tool's GUI prior to execution, while the Execute GP Tool step will immediately begin executing the tool, with no option to update parameters. NYSDOT currently uses the Launch GP Tool step type to allow for outputting an optional log file to disk and easier troubleshooting. When the Launch GP Tool step is configured as shown in the table below, the tool will have all of the necessary input parameters prepopulated with the appropriate information. The end user must simply click OK, and the tool will start to execute in the foreground.
The tool requires that all users have an SDE file in the ArcCatalog Database Connections directory with a specific name for the production_ws parameter as well as the reviewer_ws parameter. These SDE files should point to the ALRS enterprise geodatabase and the Data Reviewer enterprise geodatabase, respectively.
| Parameter | Value |
|---|---|
| job__started_date | [JOB:STARTED_DATE] |
| job__owned_by | [JOB:OWNED_BY] |
| job__id | [JOB:ID] |
| production_ws | Database Connections\dev_elrs_ad_Lockroot.sde |
| production_ws_version | ELRS.Lockroot |
| reviewer_ws | Database Connections\dev_elrs_datareviewer_dr_user.sde |
| log_path | |
| log_level | INFO |
| batch_job_file | P:\Office of Engineering\Technical Services\Highway Data Services Bureau\GIS\Roads_And_Highways_Tools\WMXDataReviewerTools\Reviewer_Batch_Jobs\RoutesInternalEventsValidations.rbj |
| full_db_flag |
The parameters of the form [JOB:FOOBAR] refer to Workflow Manager Tokens
Once the Launch or Execute GP Tool steps have been established, they can be dropped into a Workflow using Workflow Manager Administrator. The Workflow can then be assigned to a Job Type, which will allow users to create reproducible workflows on their own. Learn more about WMX in the Esri documentation.
This section of the README was updated on 2020 March 05.
This repository is currently being used in two environments at NYSDOT. The separate environments are managed using branches in this git repository. The two environments are currently the ELRS Dev environment and the temporary ELRS Prod environemnt (ArcGIS/R&H 10.5.1 with partial internal event migration to support OPPM).
The ELRS Dev environment is the current development environment for the full NYSDOT R&H deployment. This environment includes the entirety of NYSDOT's roadway inventory as internal events, and it can successfully sync external events to the AgileAssets system.
The ELRS Dev environment codebase lives in the master branch of this repository. To ensure you are currently on master, you should install a git command line interface, navigate to the root of this repository, and run the following command:
git statusIf you are on a different branch, run the following command to switch to the master branch:
git checkout -b masterThe ELRS Dev deployment uses a SQL Server SDE database for the Data Reviewer results. This means that all users will be writing their validations to the same geodatabase.
Since Roads and Highways supports route shapes that are considered invalid by the OGC, these routes will trigger validations in Data Reviewer (e.g. Invalid Shape, Duplicate Vertices). As NYSDOT upgrades to 10.7.1, we should consider a mechanism to share DR exceptions across users of the SDE DR Workspace. As it's currently configured, DR checks for duplicates in the current session. If we were to extend the workflow to delete all validations that were marked as fixed or unaddressed by the user, leaving only the exceptions behind, we can reconfigure DR to search for duplicates across the full database. This workflow would mean that if user SVC\jdoe marked a route as an exception, which was then revalidated by user SVC\jdoe2, it would not be presented as an error to SVC\jdoe2. The Data Reviewer team has developed geoprocessing tools that can support this workflow, but they only work up to ArcGIS 10.6. Esri has committed to updating these tools, so we'll need to keep an eye out for the updated package and test this workflow once we're on Desktop 10.7.1.
There was a partial migration of some select inventory elements to the NYSDOT production R&H environment to support applications deployed in some of NYSDOT's Divisions. These internal events are not actively maintained, but rather a stop-gap measure to ensure the other production systems can get the required data.
The Temporary Production environment has a different data model than ELRS Dev, thus the validations in Temp Prod are currently widdled back dramatically. The Workflow Manager Workflow in temporary prod runs the Execute Reviewer Batch Job on R&H Edits tool on the user's version, rather than the Execute All Validations tool as explained in other sections of this document.
Further, the RBJ file has been slightly modified to ensure it will work with the temporary prod environment. Rather than using the standard RoutesInternalEventsValidations.rbj file, the temporary prod deployment uses the TemporaryProdValidations.rbj file.
The Temporary Prod codebase lives in the temp-prod branch of this repository. To ensure you are currently on temp-prod, you should install a git command line interface, navigate to the root of this repository, and run the following command:
git statusIf you are on a different branch, run the following command to switch to the temp-prod branch:
git checkout -b temp-prodThe temporary prod deployment uses a file geodatabase for each individual user to store the Data Reviewer results. The file geodatabase must be in a standard file location across all users' systems, so that WMX will be able to discover it. The file geodatabases are currently stored on the D:\ drive, which is a standard drive mapped on all DOT computers. Data Reviewer is configured to look for duplicates across the full database, so users will not see duplicates that they themselves have validated. However, since the file geodatabase is local to the user's machine, there is no knowledge of colleagues validations.
Temporary prod requires a slightly different configuration in the Workflow Manager Launch GP Tool Step. The configuration should be as follows:
| Parameter Name | Parameter Value |
|---|---|
| job__started_date | [JOB:STARTED_DATE] |
| job__owned_by | [JOB:OWNED_BY] |
| job__id | [JOB:ID] |
| production_ws | Database Connections\prod_elrs_elrs_ad_Lockroot.sde |
| reviewer_ws | D:\TempProdDataReviewerWs.gdb |
| log_path | |
| log_level | INFO |
| batch_job_file | \dot-smb\dot_shared\DOT Data\Office of Engineering\Technical Services\Highway Data Services Bureau\GIS\Roads_And_Highways_Tools\DataReviewer_TemporaryProd\WMXDataReviewerTools\Reviewer_Batch_Jobs\TemporaryProdRoutesValidations.rbj |
| full_db_flag |
The easiest way to manage separate deployments is using git. This repository can be cloned onto the P drive, then you can create a new branch if required, switch to an existing branch, update the code to suit your needs, and commit the changes back to the repository. To create a new deployment, simply navigate to the folder you want to save the codebase to and run the following command:
git clone https://github.com/vitale232/WMXDataReviewerTools.git- With much of the code living in a Python package called validation_helpers, it's hard to get ArcGIS to consistently update the tool. Just open a blank map document with the Data Reviewer extension activated whenever you'd like to test code changes.
- VSCode is failing to correctly lint the validation_helpers package. It incorrectly identifies properly imported code. To workaround the issue, if you use the Open Folder function, open the ./NYSDOT_Validations_Toolbox directory instead of the project root or use a different IDE.
- When this README.md document is updated, make sure to run the
convert.jsscript in./docs/markdown-to-html-github-style-master. It's a slightly modified version of the Markdown to GitHub Style Web repository. Change into that directory, and runnode convert.js "NYSDOT Validations Toolbox README". This command will generate a static HTML document of this readme file, so that users without access to github.com will be able to view the documentation.