ProGuide Plugin Datasources

Language:      C# 7.2
Subject:       Plugin Datasource
Contributor:   ArcGIS Pro SDK Team <arcgisprosdk@esri.com>
Organization:  Esri, http://www.esri.com
Date:          04/04/2024
ArcGIS Pro:    3.3
Visual Studio: 2022

This ProGuide discusses how to build and interact with a Plugin Datasource at Pro 2.3+. Use a plugin datasource to wrap an unsupported data source type. We will be using, as our example, the Simple Point Plugin Datasource which wraps .csv formatted files. Plugin data sources are read-only

Note: Please refer to Pro Plugin registry key settings for use with Pro Plugin datasources.

Prerequisite

Download the community samples data sets from the CommunitySampleData zip. Unzip the data to a C:\Data folder. You will need the C:\Data\SimplePointData folder for the plugin datasource sample. You will also need the entire source code from the Simple Point Plugin Datasource sample. You must have Visual Studio 2022 installed to compile and build the plugin datasource sample.

Step 1

Open visual studio. Open the SimplePointPlugin.sln downloaded from the Pro SDK community samples. It contains two projects: One is a regular addin that consumes the plugin datasource: SimplePointPluginTest. The second is the plugin datasource itself: SimplePointPlugin. We will be looking at the Plugin Datasource first. When we have completed the plugin datasource implementation, we look at SimplePointPluginTest to see how to access and interact with the plugin datasource.

SimplePointPlugin uses the Microsoft.VisualBasic.FileIO.TextFieldParser to parse the csv and the "RBush" spatial index NuGet. Feel free to substitute different libraries for parsing csv files and/or spatial indexing. If the RBush NuGet does not automatically download then you will need to download and install it manually via the NuGet Package Manager in Visual Studio. Install the latest stable version of "RBush" (2.0.46 at this time of writing).

Save the project.

Step 2

Notice that the SimplePointPlugin project contains 3 source files (4 if you include the helper RBushCoord3D.cs).

• ProPluginDatasourceTemplate
• ProPluginTableTemplate
• ProPluginCursorTemplate

ProPluginDatasourceTemplate implements our custom data source wrapper, known as a PluginDatasourceTemplate. ProPluginTableTemplate derives from PluginTableTemplate and provides read/search/query capabilities of the underlying csv files(s). ProPluginCursorTemplate derives from PluginCursorTemplate and provides forward cursor access to selected rows from the ProPluginTableTemplate. The purpose of each template is described fully in ProConcepts Plugin Datasource.

Note: To create a new Plugin Datasource project, run the 'ArcGIS Pro Plugin Datasource" project template installed with the 2.3+ Pro SDK.

Step 3

First, we will examine some highlights of ProPluginDatasourceTemplate which is our implementation of the PluginDatasourceTemplate. The closest corollary to a 10x Arcobjects plugin datasource being the plug-in workspace factory helper class.

The ProPluginDatasourceTemplate provides information about its data source as well as returns plugin datasource table instances to satisfy requests for specific data sets (in this case, named csv files) from the plugin datasource. The PluginDatasourceTemplate acts as a per-thread singleton (similar to how workspace factories behave in Arcobjects). In Pro, plugin datasource data can be requested from many different threads but always from the same plugin datasource instance per thread. In other words:

• There will only be one instance of your PluginDatasourceTemplate or none per thread.
• A PluginDatasourceTemplate instance is accessed on the thread on which it was created. It will never be accessed from a different thread.
• Even though PluginDatasourceTemplate is a singleton per thread, you can have multiple instances of your PluginDatasourceTemplate "alive", one per executing thread, at any point in time. If you do share data across PluginDatasourceTemplate instances then you are responsible for managing any concurrent access.

The framework ensures that the correct ProPluginDatasourceTemplate instance is always retrieved for the given executing thread. If a ProPluginDatasourceTemplate instance has not been instantiated for a given executing thread (or has since been closed) then a new instance will be created (specifically for that thread) and its Open method called.

Open(Uri connectionPath) and Close()

Open() and Close() are called only once during the lifetime of any particular PluginDatasourceTemplate instance. Open is called when the PluginDatasourceTemplate is initialized. Close is called when the PluginDatasourceTemplate is going to be destroyed.

During Open, the PluginDatasourceTemplate is expected to perform any necessary initialization it needs in preparation of the data integration process. In the case of the SimplePointPlugin sample, the ProPluginDatasourceTemplate consumes a path to a local folder containing 0 or more csv files (which it treats as tables). The plugin datasource checks that the path exists otherwise it throws an exception and plugin datasource initialization is terminated. The plugin datasource sample uses a local cache in the form of a .NET Dictionary which is instantiated during Open.

public override void Open(Uri connectionPath) {
      //Check that the path exists
      if (!System.IO.Directory.Exists(connectionPath.LocalPath))
      {
        throw new System.IO.DirectoryNotFoundException(connectionPath.LocalPath);
      }
      ...
      ...
      _tables = new Dictionary<string, PluginTableTemplate>();//Initialize the table cache
    }

Close() is called when the PluginDatasourceTemplate is being closed and destroyed. Use Close to cleanup any internal resources such as file handles, database connections, deleting temporary files, etc. or whatever is needed to signal the end of that plugin datasource's “session”. In the sample, ProPluginDatasourceTemplate uses Close to clean up its internal table cache. Each plugin datasource table in the sample implements IDisposable as it offers a convenient pattern for clean up but it is not required (to implement IDisposable). The cleanup mechanism is at the discretion of the plugin datasource implementer and, as a minimum, any unmanaged resources should be freed during cleanup as they will not be garbage collected.

public override void Close()
    {
      //Dispose of any cached table instances here
      foreach(var table in _tables.Values)
      {
        //The sample uses IDisposable
        ((ProPluginTableTemplate)table).Dispose();
      }
      _tables.Clear();
    }

GetTableNames()

GetTableNames() should return the names of content or other data available from your plugin datasource data source. In the case of the sample, it returns the names of all csv files contained in its data folder. In simplified form, the GetTableNames() method in the sample implements code similar to this:

   public override IReadOnlyList<string> GetTableNames()
   {
      //return a list of csv file names, upper cased and without the suffix
      return
        System.IO.Directory.GetFiles(_filePath, "*.csv", 
             System.IO.SearchOption.TopDirectoryOnly)
                 .Select(
                     fn => System.IO.Path.GetFileNameWithoutExtension(fn)
                                                       .ToUpper()).ToList();
   }

If the plugin datasource wrapped a proprietary or unsupported database, for example, it would be responsible for extracting accessible data table names from the relevant data dictionary or similar. Regardless of the actual names the plugin datasource returns, the plugin datasource must ensure that the returned names can be used successfully as arguments to its PluginDatasourceTemplate.OpenTable() method.

OpenTable(string name)

On receiving an OpenTable call from the framework, the plugin datasource retrieves the requested data from its internal data source. The framework can pass in to OpenTable any string it received from GetTableNames as well as any arbitrary string provided to it by 3rd parties (such as .NET addins requesting a plugin datasource table via the ArcGIS.Core.Data API). It is the implementer's decision to decide whether to cache data internally (to improve loading performance) or not.

In the sample, ProPluginDatasourceTemplate checks its table cache (a .NET dictionary) to see if a csv file has been previously requested otherwise it reads in the csv data from disk. The plugin datasource also checks that the input "table" name is valid by checking it against its list of csv file names retrieved from GetTableNames(). If the table name is invalid it throws a GeodatabaseTableException. In simplified form, the OpenTable(string name) method in the sample is similar to:

  public override PluginTableTemplate OpenTable(string name) {
      
      var table_name = System.IO.Path.GetFileNameWithoutExtension(name).ToUpper();
      //Check table name...
      ...
      //Check if the table is in the cache...
      if (!_tables.Keys.Contains(table_name)) {
        //No - so add it
        string path = System.IO.Path.Combine(_filePath, 
                               System.IO.Path.ChangeExtension(name, ".csv"));
        _tables[table_name] = new ProPluginTableTemplate(path, 
                                         table_name, SpatialReferences.WGS84);
      }
      return _tables[table_name];

IsQueryLanguageSupported()

Return true from IsQueryLanguageSupported if your plugin datasource supports a WhereClause (typically sql) passed to the plugin datasource via a QueryFilter. Returning false, which is the default, means that any query filter where clause passed to your plugin table instance(s) will always be set to an empty string (regardless of what the client set it to).

Step 4

In this step we begin the implementation of the PluginTableTemplate. A PluginTableTemplate wraps a given data set to provide either a table or feature class to external clients. It does not necessarily need to correspond to a specific database table or dataset within the plugin datasource though whatever data it wraps will be provided to clients as a table. Therefore, regardless of how data is structured internally, a PluginTableTemplate must be able to return data to the framework "as a table" with data as rows and with each row having a unique identifier (i.e. "object id").

The details of a given plugin datasource table template implementation will be specific to the internal format of the data it wraps. In the SimplePointPlugin sample, it wraps csv formatted data. The sample parses csv data that either contains X,Y, and optionally Z data or no spatial content at all. The sample expects that the spatial data, if present, is contained in the first two (or three) columns of the csv and is point features only. The sample also assumes that the spatial data is in LAT,LONG (i.e. WGS 84).

In the sample, the requested csv is opened in the constructor of the ProPluginTableTemplate. A Microsoft.VisualBasic.FileIO.TextFieldParser parses the csv and store records internally within a System.Data.DataTable. The RBush NuGet is used to build a spatial index. A simplified version of the csv "open" is shown below:

  public class ProPluginTableTemplate : PluginTableTemplate, ... {

  private void Open() {
      //Read in the CSV
      TextFieldParser parser = new TextFieldParser(_path);
      ...
      //Initialize our data table
      _table = new DataTable();
     ...

      //read the csv line by line
      while (!parser.EndOfData) {
        var values = parser.ReadFields();
        ...

        //load the datatable
        var row = _table.NewRow();
        for (int c = 0; c < values.Length; c++)
          row[c + 1] = values[c] ?? "";//Column "0" is our objectid
        ...
        //get the coordinate, if we have one
        var coord = new Coordinate3D(x, y, z);
            
        //store it in the DataTable row
        row["SHAPE"] = coord.ToMapPoint().ToEsriShape();

        //add it to the index
        var rbushCoord = new RBushCoord3D(coord, (long)row["OBJECTID"]);
        _rtree.Insert(rbushCoord);
        _table.Rows.Add(row);
      }

Step 5

The next step is to implement the PluginTableTemplate capabilities. A PluginTableTemplate returns "properties" about its stored data (a name, what fields, extent, and so forth) and provides search results (in the form of a cursor) in response to queries. Let's look at the "property" information first.

GetName()

Return the name of the plugin datasource table. It should correspond to the same name returned from the plugin datasource GetTableNames().

GetShapeType()

Return either a supported GeometryType or GeometryType.Unknown if there is no SHAPE type. Plugin datasource tables returning a valid GeometryType are treated as feature classes. Plugin datasource tables returning GeometryType.Unknown are treated as tables only.

GetExtent()

If your plugin datasource returns a valid GeometryType, it must return an extent (even if the table is empty). The framework uses the extent to determine:

• The feature class spatial reference (accessed off ArcGIS.Core.Data.FeatureClassDefinition)
• HasZ (ditto)
• HasM (ditto)

In addition, the framework is using GetExtent() to determine the 'Full Extent' and 'Default Extent' of the plugin datasource. The extent returned by GetExtent() is used by functions like ‘Zoom to Layer’ or setting the initial extent when a plugin datasource is added to a map.

GetLastModifiedTime()

GetLastModifiedTime() returns the timestamp of when the actual data that feeds the plugin datasource was last modified. The default is the minimum possible date System.DateTime.MinValue. You can use GetLastModifiedTime() to update the time of the last modification (add/delete/update) to the actual datasource. When GetLastModifiedTime() returns a newer timestamp, the framework will refresh the full/default extent returned by GetExtent() for the plugin datasource layer. You have to update the value returned by GetExtent() before you change the value returned by GetLastModifiedTime(). GetLastModifiedTime() was added in Release 3.1, before this addition the value returned by GetExtent() was used for the duration of the session.

IsNativeRowCountSupported() and GetNativeRowCount()

These methods exist strictly for performance reasons. If the framework needs a count of all the rows in a plugin datasource table it will call GetNativeRowCount() to retrieve the number of rows if the plugin datasource table returns true from IsNativeRowCountSupported(). If IsNativeRowCountSupported() returns false (the default) then the framework will call Search() on the plugin datasource table and will manually iterate through the cursor to count the number of rows.

In the case of the sample, it can use either the spatial index collection or the underlying data table to get the count of rows without the need for a query. Therefore the ProPluginTableTemplate IsNativeRowCountSupported implementation returns true.

GetFields()

GetFields must return a collection of ArcGIS.Core.Data.PluginDatastore.PluginFields for all fields available in the plugin datasource table. Plugin datasource tables that support spatial data must include a PluginField of field type ArcGIS.Core.Data.FieldType.Geometry to be considered a feature class. GetFields must also return the fields in the exact same order as fields will occur in rows retrieved via a Search.

This is the GetFields implementation from the SimplePointPlugin sample.

  public override IReadOnlyList<PluginField> GetFields() {
      var pluginFields = new List<PluginField>();
      foreach (var col in _table.Columns.Cast<DataColumn>())
      {
        var fieldType = ArcGIS.Core.Data.FieldType.String;
        //special handling for OBJECTID and SHAPE
        if (col.ColumnName == "OBJECTID")
          fieldType = ArcGIS.Core.Data.FieldType.OID;
        
        else if (col.ColumnName == "SHAPE")
          fieldType = ArcGIS.Core.Data.FieldType.Geometry;
        
        pluginFields.Add(new PluginField() {
          Name = col.ColumnName,
          AliasName = col.Caption,//Alias name is required
          FieldType = fieldType
        });
      }
      return pluginFields;
    

Note: PluginField.AliasName cannot be left blank. If a column does not have an alias name then set the alias name to be the same as the column name.

If the column has a length property in the underlying datastore, it can be assigned to the PluginField.Length property to set the field’s maximum number of characters.

var pluginField = new PluginField() {
  Name = col.ColumnName,
  AliasName = col.Caption,
  FieldType = fieldType,
  Length = col.ColumnLength
}

By default, the maximum number of characters allowed for a text field is 255.

Step 6

Plugin datasource tables must implement Search. If the parent plugin datasource returned true from PluginDatasourceTemplate.IsQueryLanguageSupported() then you must implement support for query filter where clauses. If your plugin datasource table contains a shape column then you must also implement support for Search with a SpatialQueryFilter.

Consult the SimplePointPlugin sample for an example implementation of Search(QueryFilter queryFilter) and Search(SpatialQueryFilter spatialQueryFilter). The guidelines for implementing Search for QueryFilter and SpatialQueryFilter are as follows:

QueryFilter

• An empty query should return all rows.
• Selection via QueryFilter.ObjectIDs must be supported. All "rows" (however your data is structured internally) with corresponding identifiers to those provided in the query filter ObjectIDs list must be selected.
• If your datasource returned true from PluginDatasourceTemplate.IsQueryLanguageSupported() implement row selection for the provided QueryFilter.WhereClause. If your datasource returned false, the WhereClause will always be empty.
• Support for QueryFilter.PrefixClause and QueryFilter.PostfixClause is optional and depends on the extent to which prefix and postfix clauses are supported by your plugin datasource's underlying data structure.
• If a SubFields clause is specified then only those values for fields listed in the QueryFilter.SubFields should be populated into the returned rows. Note: a returned row must still contain all fields (not just the fields in the SubFields clause) and the ordering of fields must match the ordering of fields returned from PluginTableTemplate.GetFields(). Field values not specified in the SubFields clause should be set to null.
• If a QueryFilter.OutputSpatialReference is provided, shapes in returned rows must be projected into the output spatial reference.
• If QueryFilter.ObjectIDs and QueryFilter.WhereClause are both set then the selected rows must be the intersection of both individual selections (i.e. "selection via object id" AND "selection by where clause").

Note: The query filter passed to Search will never be null.

SpatialQueryFilter

• Implement support for all the guidelines specified for QueryFilter and...
• If a SpatialQueryFilter.FilterGeometry is provided, the plugin datasource should perform a spatial selection using the filter geometry. The filter geometry is evaluated using the provided SpatialQueryFilter.SpatialRelationship.
• SpatialQueryFilter.SearchOrder, if specified, can be used by the plugin datasource to determine whether to execute the where clause query first or the spatial query with the filter geometry first (and the other one second).
• The results of the spatial query filter must be intersected ("and-ed") with any results from the underlying query filter (the object ids list, if there was one, intersected with the where clause, if there was one).

Note: SpatialQueryFilter will never be null. Search(SpatialQueryFilter) will never be called on your plugin datasource table instance if it did not return a shape field from GetFields(). A SpatialQueryFilter.SpatialRelationship will always be provided if a SpatialQueryFilter.FilterGeometry is provided. Values of SpatialRelationship.Undefined will be caught by the framework (which will throw an ArgumentException).

Step 7

The plugin datasource table is responsible for creating a PluginCursorTemplate that contains, or accesses, the results of the given search. PluginCursorTemplate is a forward-only, read-only, cursor that will be used to "move" forward, row-by-row, over the selected rows. The PluginCursorTemplate is responsible for:

• Traversing the internal data structure of the plugin datasource table to return the Search results as ArcGIS.Core.Data.PluginDatastore.PluginRow instances.
• Keeping track of its position within the set of selected rows (or whatever the internal data structure it is traversing is)

Calls to PluginCursorTemplate.MoveNext() advance the cursor to the next row in the set. The next row is made available via PluginCursorTemplate.GetCurrentRow(). A call to MoveNext will always precede the call to GetCurrentRow. Calls to MoveNext and GetCurrentRow are always on the same thread that created the plugin datasource table (and its parent plugin datasource). When the PluginCursorTemplate is advanced passed the end of its selected internal set (via a MoveNext) it must return false. The behavior of a call made to GetCurrentRow after MoveNext failed (returned false) is undefined. Ideally it should return null.

In the SimplePointPlugin sample, the ProPluginCursorTemplate maintains its set of selected row ids as a .NET queue. On each MoveNext call, the next id is popped from the queue until there are no more ids left in which case MoveNext returns false. In the sample, the plugin datasource table provides the row information for the current id to the cursor whenever ProPluginCursorTemplate.GetCurrentRow() is called. In simplified form, the implementation of the cursor in the sample is:

public class ProPluginCursorTemplate : PluginCursorTemplate {

    private Queue<long> _oids;
    private long _current = -1;

    public override PluginRow GetCurrentRow() {
      ...
      return _provider.FindRow(_current, ...);//"provider" is the plugin datasource table
    }

    public override bool MoveNext() {
      if (_oids.Count == 0)
        return false;
      ...
      _current = _oids.Dequeue();
      return true;
    }

Step 8

Your completed plugin datasource can be deployed to client machines using the same approach as an addin (in Pro, plugin data sources are addins). Double click the .esriPlugin archive to have RegisterAddin.exe install it to your default user folder. Similarly, plugin datasources can be copied to well-known folders (again, same as "regular" Pro addins). More detail on the structure of a plugin datasource can be found in the Framework ProConcepts guide in the Plugin datasource section.

Note: Installed plugin datasources are not listed in the Addin Manager tab on the backstage of Pro.

Step 9

Plugin data sources are always delay-loaded. They are not instantiated until a client opens the plugin data source and starts to request data. The plugin datasource framework ensures that the plugin data source and its contained plugin datasource tables appear as feature classes and/or tables to both the native c++ code of Pro as well as to .NET clients.

For 3rd party developers, a custom plugin data source is initialized through the ArcGIS.Core.Data api and follows a similar pattern as would be used to access other Datastores such as a file GDB or enterprise database. Assuming the plugin data source has been deployed to the relevant machine, You will need two pieces of information:

• The plugin data source id (from the plugin datasource Config.xml)
• The data source path uri (a path or url to the custom data set the plugin datasource should load or to a file containing the relevant connection information for the data source)

The plugin data source id is stored in the plugin datasource Config.xml. The plugin datasource id uniquely identifies the plugin datasource on the given system and is used by Pro to identify which plugin data source it needs to load.

<?xml version="1.0" encoding="utf-8"?>
<ESRI.Configuration xmlns="http://schemas.esri.com/DADF/Registry" ...>
  <Name>Example SimplePointPlugin Datasource</Name>
  ...
  
  <AddIn language="CLR4.6.1" library="SimplePointPlugin.dll" namespace="...>
    <ArcGISPro>
      <PluginDatasources>
        <PluginDatasource id="SimplePointPlugin1_Datasource" class="... />
      </PluginDatasources>
    

In the code snippet below, an instance of ArcGIS.Core.Data.PluginDatastore.PluginDatastore is created with the ArcGIS.Core.Data.PluginDatastore.PluginDatasourceConnectionPath that has the plugin datasource id and data source path that should be used. The data source path is passed to the Open(Uri connectionPath) method of the underlying PluginDatasourceTemplate (which we covered in Step 3).

  await QueuedTask.Run(() =>  {
        using (var pluginws = new PluginDatastore(
             new PluginDatasourceConnectionPath("SimplePointPlugin1_Datasource",
                   new Uri(@"C:\Data\SimplePointData", UriKind.Absolute))))  {

          //TODO - use the plugin data source
          ...

Calling OpenTable() on the PluginDatastore results in a call to OpenTable() on the PluginDatasourceTemplate (which we covered in Step 3 also). A PluginTableTemplate instance will be instantiated and returned to the client in the form of a feature class or table. The framework provides the client with the ArcGIS.Core.Data.Dataset (feature class or table) that uses the plugin datasource table.

   foreach (var table_name in pluginws.GetTableNames()) {

         var feat_class = pluginws.OpenTable(table_name) as FeatureClass;

SimplePointPluginTest addin
The SimplePointPlugin sample is paired with a Module Add-in project, SimplePointPluginTest (in the same SimplePointPlugin solution).

Note: Starting with release 2.8, if you add a Module Add-in project to a Visual Studio solution that already contains a Plug-in Datasource project a 'build all' action on the solution skips the Module Add-in during the build process. To fix this issue, open the solution's 'Configuration Manager' dialog and check the build check box for the Module Add-in project.

SimplePointPluginTest illustrates two different usage patterns for interacting with a plugin datasource. There are two buttons: TestCsv1.cs and TestCsv2.cs:

TestCsv1
TestCsv1 illustrates interacting with the plugin datasource exclusively via the ArcGIS.Core.Data api. The code retrieves a feature class from the the plugin data source, queries it for its ArcGIS.Core.Data.FeatureClassDefinition and queries it for its rows via Table.Search(). The returned row cursor is manipulated (same as any standard ArcGIS.Core.Data.RowCursor) to retrieve the selected rows. Set breakpoints in the addin and plugin datasource to observe how the client and plugin datasource interact.

  await QueuedTask.Run(() =>  {

        using (PluginDatastore pluginws = new PluginDatastore(
             new PluginDatasourceConnectionPath("SimplePointPlugin1_Datasource",
                   new Uri(@"C:\Data\SimplePointData", UriKind.Absolute)))) {
          
          foreach (var table_name in pluginws.GetTableNames()) {
            
            using (var table = pluginws.OpenTable(table_name))
            {
              //get information about the table
              using(var def = table.GetDefinition() as FeatureClassDefinition) 
              ...

              //query and return all rows
              using(var rc = table.Search(null)) {
                while(rc.MoveNext()) {
                  var feat = rc.Current as Feature;
                  

TestCsv2
TestCsv2 illustrates another common usage pattern for plugin datasources. It retrieves all the feature classes from the plugin data source and adds each one as the data source to a new Layer (which are added to the active map or scene). At that point the .NET addin code is "out of the loop". Once in the TOC, Pro itself interacts with the underlying plugin datasource (via the feature layers) for purposes of pan, zoom, select, identify, etc. You can also open an attribute table to list the plugin datasource table records and open the symbology tab to change each layer's default symbology. If the project is saved, the next time the relevant map or scene is opened, it will hydrate the plugin data source automatically without the need for any intervention from the .NET SimplePointPluginTest addin.

 await QueuedTask.Run(() => {
        using (var pluginws = new PluginDatastore(
             new PluginDatasourceConnectionPath("SimplePointPlugin1_Datasource",
                   new Uri(@"C:\Data\SimplePointData", UriKind.Absolute))))  {

          foreach (var table_name in pluginws.GetTableNames()) {
            using (var table = pluginws.OpenTable(table_name))  {
              //Add as a layer to the active map or scene
              LayerFactory.Instance.CreateFeatureLayer((FeatureClass)table, 
                                                                 MapView.Active.Map);
            }