diff --git a/docs/docs/Building_Your_First_Plugin/Data_Ingestion.md b/docs/docs/Building_Your_First_Plugin/Data_Ingestion.md index 87c04dfa..a3aaf6d3 100644 --- a/docs/docs/Building_Your_First_Plugin/Data_Ingestion.md +++ b/docs/docs/Building_Your_First_Plugin/Data_Ingestion.md @@ -21,7 +21,7 @@ quite limiting. For our first plugin, we will be using the more flexible [staging](/References/Glossary.md#staged-linkingsyncing) strategy. With this strategy, the Delphix Engine uses NFS for Unix environments (or iSCSI on Windows environments) to mount storage onto a [staging environment](/References/Glossary.md#staging-environment). Our plugin will then be in full control of how to get data from the source environment onto this storage mount. -With the staging strategy, there are two types of syncs: sync and resync. A `sync` is used to ingestion incremental changes while a `resync` is used to re-ingest all the data for the dSource. For databases, this could mean re-ingesting from a full database backup to reset the dSource. A `sync` and a `resync` execute the same plugin operations and are differentiated by a boolean flag in the [snapshot_parameters](/References/Classes.md#snapshotparametersdefinition) argument passed into [linked.pre_snapshot](/References/Plugin_Operations.md#staged-linked-source-pre-snapshot) and [linked.post_snapshot](/References/Plugin_Operations.md#staged-linked-source-post-snapshot). +With the staging strategy, there are two types of syncs: sync and resync. A `sync` is used to ingest incremental changes while a `resync` is used to re-ingest all the data for the dSource. For databases, this could mean re-ingesting from a full database backup to reset the dSource. A `sync` and a `resync` will execute the same plugin operations. To differentiate a `sync` from a `resync`, simply add a boolean property (i.e. `resync`) in the plugin's [snapshot parameters definition](References/Schemas_and_Autogenerated_Classes.md#snapshotparametersdefinition-schema). Once `sync` or `resync` is selected, the property will be passed into [linked.pre_snapshot](/References/Plugin_Operations.md#staged-linked-source-pre-snapshot) and [linked.post_snapshot](/References/Plugin_Operations.md#staged-linked-source-post-snapshot) as a [snapshot parameter](/References/Glossary.md#snapshot-parameters). A regular `sync` is the default and is executed as part of policy driven syncs. A `resync` is only executed during initial ingestion or if the Delphix user manually starts one. The customer can manually trigger a `resync` via the UI by selecting the dSource, going to more options and selecting **Resynchronize dSource**. ![Screenshot](images/Resync.png) @@ -168,7 +168,7 @@ Next, we'll add a new function: ```python @plugin.linked.pre_snapshot() -def copy_data_from_source(staged_source, repository, source_config, snapshot_parameters): +def copy_data_from_source(staged_source, repository, source_config, optional_snapshot_parameters): stage_mount_path = staged_source.mount.mount_path data_location = "{}@{}:{}".format(staged_source.parameters.username, staged_source.parameters.source_address, @@ -249,7 +249,7 @@ In fact, the default implementation that was generated by `dvp init` will work j def linked_post_snapshot(staged_source, repository, source_config, - snapshot_parameters): + optional_snapshot_parameters): return SnapshotDefinition() ``` diff --git a/docs/docs/References/Classes.md b/docs/docs/References/Classes.md index 07097a22..afcd2850 100644 --- a/docs/docs/References/Classes.md +++ b/docs/docs/References/Classes.md @@ -144,29 +144,6 @@ Field | Type | Description mounts | list[[Mount](#mount)] | The list of mounts to export the data sets to. ownership_specification | [OwnershipSpecification](#ownershipspecification) | **Optional.** Control the ownership attributes for the data set. It defaults to the environment user of the remote environment if it is not specified. -## SnapshotParametersDefinition - -User provided parameters for the snapshot operation. It includes a boolean property named `resync` that can be used to indicate to the plugin whether or not to initiate a full ingestion of the dSource. The parameters are only set during a manual snapshot. When using a sync policy, `resync` defaults to `false`. - -```python -from dlpx.virtualization.platform import Plugin - -plugin = Plugin() - -@plugin.linked.pre_snapshot() -def linked_pre_snapshot(staged_source, repository, source_config, snapshot_parameters): - if snapshot_parameter.resync: - print(snapshot_parameter.resync) -``` - -> This class will be generated during build and is located with the autogenerated classes. As it is passed into the operation, importing it is not neccessary. - -### Fields - -Field | Type | Description ------ | ---- | ----------- -resync | Boolean | Determines if this snapshot should ingest the dSource from scratch. - ## RemoteEnvironment Represents a remote environment. diff --git a/docs/docs/References/Plugin_Operations.md b/docs/docs/References/Plugin_Operations.md index 2fe76756..e02e3773 100644 --- a/docs/docs/References/Plugin_Operations.md +++ b/docs/docs/References/Plugin_Operations.md @@ -177,7 +177,7 @@ Sets up a [dSource](Glossary.md#dsource) to ingest data. Only applies when using ### Signature -`def linked_pre_snapshot(direct_source, repository, source_config)` +`def linked_pre_snapshot(direct_source, repository, source_config, optional_snapshot_parameters)` ### Decorator @@ -190,6 +190,7 @@ Argument | Type | Description direct_source | [DirectSource](Classes.md#directsource) | The source associated with this operation. repository | [RepositoryDefinition](Schemas_and_Autogenerated_Classes.md#repositorydefinition-class) | The repository associated with this source. source_config | [SourceConfigDefinition](Schemas_and_Autogenerated_Classes.md#sourceconfigdefinition-class) | The source config associated with this source. +optional_snapshot_parameters | [SnapshotParametersDefinition](Classes.md#snapshotparametersdefinition) | The snapshot parameters. The value is `None` when executed during a snapshot policy. ### Returns None @@ -220,7 +221,7 @@ Captures metadata from a [dSource](Glossary.md#dsource) once data has been inges ### Signature -`def linked_post_snapshot(direct_source, repository, source_config)` +`def linked_post_snapshot(direct_source, repository, source_config, optional_snapshot_parameters)` ### Decorator @@ -233,6 +234,7 @@ Argument | Type | Description direct_source | [DirectSource](Classes.md#directsource) | The source associated with this operation. repository | [RepositoryDefinition](Schemas_and_Autogenerated_Classes.md#repositorydefinition-class) | The repository associated with this source. source_config | [SourceConfigDefinition](Schemas_and_Autogenerated_Classes.md#sourceconfigdefinition-class) | The source config associated with this source. +optional_snapshot_parameters | [SnapshotParametersDefinition](Classes.md#snapshotparametersdefinition) | The snapshot parameters. The value is `None` when executed during a snapshot policy. ### Returns [SnapshotDefinition](Schemas_and_Autogenerated_Classes.md#snapshotdefinition-class) @@ -246,7 +248,7 @@ from generated.definitions import SnapshotDefinition plugin = Plugin() @plugin.linked.post_snapshot() -def linked_post_snapshot(direct_source, repository, source_config): +def linked_post_snapshot(direct_source, repository, source_config, optional_snapshot_parameters): snapshot = SnapshotDefinition() snapshot.transaction_id = 1000 return snapshot @@ -277,7 +279,7 @@ Sets up a [dSource](Glossary.md#dsource) to ingest data. Only applies when using ### Signature -`def linked_pre_snapshot(staged_source, repository, source_config, snapshot_parameters)` +`def linked_pre_snapshot(staged_source, repository, source_config, optional_snapshot_parameters)` ### Decorator @@ -290,7 +292,7 @@ Argument | Type | Description staged_source | [StagedSource](Classes.md#stagedsource) | The source associated with this operation. repository | [RepositoryDefinition](Schemas_and_Autogenerated_Classes.md#repositorydefinition-class) | The repository associated with this source. source_config | [SourceConfigDefinition](Schemas_and_Autogenerated_Classes.md#sourceconfigdefinition-class) | The source config associated with this source. -snapshot_parameters | [SnapshotParametersDefinition](Classes.md#snapshotparametersdefinition) | The snapshot parameters. +optional_snapshot_parameters | [SnapshotParametersDefinition](Classes.md#snapshotparametersdefinition) | The snapshot parameters. The value is `None` when executed during a snapshot policy. ### Returns None @@ -303,7 +305,7 @@ from dlpx.virtualization.platform import Plugin plugin = Plugin() @plugin.linked.pre_snapshot() -def linked_pre_snapshot(staged_source, repository, source_config, snapshot_parameters): +def linked_pre_snapshot(staged_source, repository, source_config, optional_snapshot_parameters): pass ``` @@ -320,7 +322,7 @@ Captures metadata from a [dSource](Glossary.md#dsource) once data has been inges ### Signature -`def linked_post_snapshot(staged_source, repository, source_config, snapshot_parameters)` +`def linked_post_snapshot(staged_source, repository, source_config, optional_snapshot_parameters)` ### Decorator @@ -333,7 +335,7 @@ Argument | Type | Description staged_source | [StagedSource](Classes.md#stagedsource) | The source associated with this operation. repository | [RepositoryDefinition](Schemas_and_Autogenerated_Classes.md#repositorydefinition-class) | The repository associated with this source. source_config | [SourceConfigDefinition](Schemas_and_Autogenerated_Classes.md#sourceconfigdefinition-class) | The source config associated with this source. -snapshot_parameters | [SnapshotParametersDefinition](Classes.md#snapshotparametersdefinition) | The snapshot parameters. +optional_snapshot_parameters | [SnapshotParametersDefinition](Classes.md#snapshotparametersdefinition) | The snapshot parameters. The value is `None` when executed during a snapshot policy. ### Returns [SnapshotDefinition](Schemas_and_Autogenerated_Classes.md#snapshotdefinition-class) @@ -347,9 +349,9 @@ from generated.definitions import SnapshotDefinition plugin = Plugin() @plugin.linked.post_snapshot() -def linked_post_snapshot(staged_source, repository, source_config, snapshot_parameters): +def linked_post_snapshot(staged_source, repository, source_config, optional_snapshot_parameters): snapshot = SnapshotDefinition() - if snapshot_parameters.resync: + if optional_snapshot_parameters is not None and optional_snapshot_parameters.resync: snapshot.transaction_id = 1000 else: snapshot.transaction_id = 10 diff --git a/docs/docs/Release_Notes/3.0.0/3.0.0.md b/docs/docs/Release_Notes/3.0.0/3.0.0.md index f6854791..e8c41e89 100644 --- a/docs/docs/Release_Notes/3.0.0/3.0.0.md +++ b/docs/docs/Release_Notes/3.0.0/3.0.0.md @@ -3,11 +3,25 @@ To install or upgrade the SDK, refer to instructions [here](/Getting_Started.md#installation). ## New & Improved -* Added a `scratch_path` property on the [RemoteHost](/References/Classes/#remotehost) object for storage and debugging purposes. +* Added the ability to define snapshot parameters in a [Snapshot Parameters Definition](/References/Schemas_and_Autogenerated_Classes.md#snapshotparametersdefinition-schema). + * Provide end-users with configurable options prior to taking a snapshot. + * The options selected are provided as input to pre/post-snapshot functions. + +* Added a `scratch_path` property on the [RemoteHost](/References/Classes/#remotehost) object which can be used as: + * A location to store small amounts of persistent data. + * A location to mount VDB data. More details about `scratch_path` can be found [here](/Best_Practices/Scratch_Paths.md) ## Breaking Changes * Added a new required schema [Snapshot Parameters Definition](/References/Schemas_and_Autogenerated_Classes.md#snapshotparametersdefinition-schema). -[**For more information and detailed steps to detect and make changes.**](/Release_Notes/3.0.0/3.0.0_Breaking_Changes#new-required-schema) \ No newline at end of file +[**For more information and detailed steps to detect and make changes.**](/Release_Notes/3.0.0/3.0.0_Breaking_Changes#new-required-schema) + +* Added a new parameter to the [Direct Linked Source Pre-Snapshot](/References/Plugin_Operations/#direct-linked-source-pre-snapshot) and [Direct Linked Source Post-Snapshot](/References/Plugin_Operations/#direct-linked-source-pre-snapshot) plugin operations. + +[**For more information and detailed steps to detect and make changes.**](/Release_Notes/3.0.0/3.0.0_Breaking_Changes#new-parameter-in-direct-prepost-snapshot-functions) + +* Renamed a parameter in the [Staged Linked Source Pre-Snapshot](/References/Plugin_Operations/#staged-linked-source-pre-snapshot) and [Staged Linked Source Post-Snapshot](/References/Plugin_Operations/#staged-linked-source-post-snapshot) plugin operations. + +[**For more information and detailed steps to detect and make changes.**](/Release_Notes/3.0.0/3.0.0_Breaking_Changes#parameter-renamed-in-staged-prepost-snapshot-functions) diff --git a/docs/docs/Release_Notes/3.0.0/3.0.0_Breaking_Changes.md b/docs/docs/Release_Notes/3.0.0/3.0.0_Breaking_Changes.md index b0c13762..7bfbddaa 100644 --- a/docs/docs/Release_Notes/3.0.0/3.0.0_Breaking_Changes.md +++ b/docs/docs/Release_Notes/3.0.0/3.0.0_Breaking_Changes.md @@ -1,7 +1,7 @@ # Breaking Changes - v.3.0.0 ## New required schema -[Snapshot Parameters Definition](/References/Schemas_and_Autogenerated_Classes.md#snapshotparametersdefinition-schema) allows plugin authors to define [Snapshot Parameters] which can be displayed to an end-user whenever a linked source snapshot is taken. +[Snapshot Parameters Definition](/References/Schemas_and_Autogenerated_Classes.md#snapshotparametersdefinition-schema) allows plugin authors to define [snapshot parameters](/References/Glossary.md#snapshot-parameters) which can be displayed to an end-user whenever a linked source snapshot is taken. ### What is affected All plugins built with v2.0.0 or below will be affected. The [Schema](/References/Schemas) must contain a `snapshotParametersDefinition`. @@ -34,3 +34,81 @@ Example: "type": "object" } ``` + +## New Parameter in Direct Pre/Post-Snapshot Functions +`optional_snapshot_parameters` has been added as a parameter in [Direct Linked Source Pre-Snapshot](/References/Plugin_Operations/#direct-linked-source-pre-snapshot) and [Direct Linked Source Post-Snapshot](/References/Plugin_Operations/#direct-linked-source-post-snapshot). + +### What is affected +All direct plugins built with v2.1.0 or below will be affected. + +### How does it fail +[dvp build](/References/CLI.md#build) will fail with the following error message if the `optional_snapshot_parameters` is not added: + +```bash +$ dvp build +Error: Named argument mismatch in method linked_post_snapshot. Expected: ['staged_source', 'repository', 'source_config', 'optional_snapshot_parameters'], Found: ['staged_source', 'repository', 'source_config']. + +0 Warning(s). 1 Error(s). + +BUILD FAILED. +``` + +### How to fix it +Add `optional_snapshot_parameters` as a parameter in [Direct Linked Source Pre-Snapshot](/References/Plugin_Operations/#direct-linked-source-pre-snapshot) and [Direct Linked Source Post-Snapshot](/References/Plugin_Operations/#direct-linked-source-post-snapshot). + +* Previous releases + +```python + @plugin.linked.post_snapshot() + def linked_post_snapshot(direct_source, repository, source_config): + return SnapshotDefinition() +``` + +* 3.0.0 + +```python + @plugin.linked.post_snapshot() + def linked_post_snapshot(direct_source, repository, source_config, optional_snapshot_parameters): + return SnapshotDefinition() +``` + +## Parameter Renamed in Staged Pre/Post-Snapshot Functions +The following parameter was renamed in the [Staged Linked Source Pre-Snapshot](/References/Plugin_Operations/#staged-linked-source-pre-snapshot) and [Staged Linked Source Post-Snapshot](/References/Plugin_Operations/#staged-linked-source-post-snapshot) functions: + +| Previous | Updated | +| -------- | ------- | +| `snapshot_parameters` | `optional_snapshot_parameters` | + +### What is affected +All staged plugins built with v2.1.0 or below will be affected. + +### How does it fail +[dvp build](/References/CLI.md#build) will fail with the following error message if the parameter is not renamed from `snapshot_parameters` to `optional_snapshot_parameters`: + +```bash +$ dvp build +Error: Named argument mismatch in method linked_post_snapshot. Expected: ['staged_source', 'repository', 'source_config', 'optional_snapshot_parameters'], Found: ['staged_source', 'repository', 'source_config', 'snapshot_parameters']. + +0 Warning(s). 1 Error(s). + +BUILD FAILED. +``` + +### How to fix it +Rename `snapshot_parameters` to `optional_snapshot_parameters` in [Staged Linked Source Pre-Snapshot](/References/Plugin_Operations/#staged-linked-source-pre-snapshot) and [Staged Linked Source Post-Snapshot](/References/Plugin_Operations/#staged-linked-source-post-snapshot). + +* Previous releases + +```python + @plugin.linked.post_snapshot() + def linked_post_snapshot(staged_source, repository, source_config, snapshot_parameters): + return SnapshotDefinition() +``` + +* 3.0.0 + +```python + @plugin.linked.post_snapshot() + def linked_post_snapshot(staged_source, repository, source_config, optional_snapshot_parameters): + return SnapshotDefinition() +``` \ No newline at end of file