Publish and support an official Extensibility API with conformance rules

[phaumer]

As a Zowe Explorer Extender I want to import a well-defined and documented extensibility API that I can use to build my VS Code extension and that I can then certify as Zowe compliant to provide value-add capabilities to Zowe Explorer.

To support a set of well-defined extensibility APIs and compliance rules for extenders the following work packages need to be completed:

1. Refine scope of Extensibility API and fix issue and implement high priority enhancements.
2. Extract API code into its own NPM package for extenders to include.
3. Define Extensibility API governance for Zowe Explorer contributors.
4. Define Extensibility Conformance rules for extenders.

1. Refine scope of Extensibility API and fix issue and implement high priority enhancements

Zowe Explorer already provides several ways for extenders to extend it.

• The api source folder provides in ZoweExplorerApi interfaces and a registry for extenders to implement the z/OS interaction for MVS, USS, JES using alternative protocols such as FTP or SSH.
• The api source folder provides in ZoweExplorerApi an interface IApiExplorerExtender and a default implementation in ZoweExplorerExtender for extenders to access cached profiles and profile links.
• The README-Extending.md describes a mechanism for providing Context Hooks for extending the current Zowe Explorer tree views with additional context menus.

In addition to the current extensibility mechanism there a list of shortcomings or issues to be solved for Extenders:

• (Issue Secure Credentials support for Extenders #1161) Secure Credentials support for Extenders: If the secure credentials CLI plugin is enabled then extension using the APIs listed above will not work as required dependencies such as keytar are loaded into their context.
• (Issue Zowe Explorer activation does not consider other extenders registering their extension after activation #742) Extension-friendly activation: Activation of Zowe Explorer is not considering potential installed extensions that get activated only after Zowe Explorer is activated, because of the VS Code Extension Dependency that determines the activation order. The Zowe Explorer activation loads only z/OSMF default profiles, even if they were hidden previously, and ignores any profiles used by extensions. It also invalidates Favorites created by extenders as these are not activated, yet.
• (Issue Extender API for Profile folder creation with meta-file #1254) Extension-friendly activation part 2: Allow extensions to not require their Profiles' CLI plugin for folder and meta file creation within .zowe/profiles.
• (Issue Standardize Zowe Explorer configuration property names to make them consistent with VS Code's format #1263) Standardize Zowe Explorer configuration property names to make them consistent with VS Code's format.
• (Issue Zowe Explorer unit test suite to review extenders' API methods #1278) Zowe Explorer unit test suite to review extenders' API methods: Facilitate verification of Zowe Explorer extenders conformance for both extenders and reviewers.
• (Issue Standardize Zowe Explorer context menus and document for extenders #1279) Standardize Zowe Explorer context menus and document for extenders.
• (Issue Make the profilesCache available to extenders #1281) Make the profilesCache available to extenders: Add common Profiles Cache to API.
• (Issue Update Zowe Explorer Extender API docs for IApiRegisterClient #1283) Update Zowe Explorer Extender API docs for IApiRegisterClient: Provide more details on implementation and testing.
• (Issue Centralized error handling #388) Centralized error handling: Localizable error handling that extenders can access.

There is also a list of enhancements and refinements extenders have already requested:

• (Issue #ZoweExplorerApi getTsoApi() and interface ITso is not implemented? #684) Add TSO API In addition to already existing API for MVS, USS, JES
• (Issue Could ZoweExplorerApi.IMvs also have "Get.dataSet()" ? #693) Add more API methods: For example there is a request for adding Get.dataSet() so extenders can stream data into their own buffers.
• (Issue Profile linking. Refine the Dialog based on availability #730) Profile linking refinements: The profile types drop-down does not always show the relevant list of types. (See issue Remove linked profiles in favor of base profiles #1280.)
• (Issue Remove linked profiles in favor of base profiles #1280) Remove linked profiles: The linked profiles capability has been superseded by base profiles.
• (Issues API extension for alternative file management and rendering (Callbacks) #601/Provide extension points for additional file handling profiles #602) File handling extension point: In addition to only allow providing alternative implementations of the file access protocol provide the means to contribute all new capabilities into the tree browsers or override/hide existing behavior.
• (Issue Can getQualifedResourceName() be added to extender API ? #745) More method to access node meta-data Add getQualifedResourceName() method to API.
• (Issue Extenders- Opening a Favorite data set results in a document path temp/_D_/Favorites/dataset(member) - the actual profile name is not included  #778) Favorites uris do not contain profile name Need to treat files opened via favorites in a way that extenders can find the profile that loaded it.
• (Issue Please can the session node contextValue include the type of the connection ( "zosmf" / "rse" / "ftp" ) #691/Can data set tree node contextValue include the "extension" or data set "type"? #686) Expose profile type to node contextValue: Then menu could be added via package.json based on profile type.
• (Issue Provide an API for appending local file extensions for data set members #1193) Provide an API for extenders to define the file extensions to be used for data set members.

2. Extract API code into its own NPM package for extenders to include

• Issue Create Zowe Explorer API as separate NPM package. #671: Extenders currently need to physically copy some Interface files from Zowe Extender to implement them. We should provide these as a separate NPM package on npmjs.org so extenders can just define a simple dependency in their codebase.

That npm package should also include all code that makes sense for extenders to reuse such as loading profiles from the file system, loading imperative settings files, and loading keytar dependencies to enable using the secure credentials plugin.

3. Define Extensibility API governance for Zowe Explorer contributors

In addition to the documentation of the API we also need to established rules for governance of the APIs for Zowe Explorer developers for changing the API. We should use the rules of semantic versioning and apply them to our APIs. For example,

• Bug fix level version must not change APIs at all
• Minor versions can add new APIs, but must add them as optional APIs that extenders are not forced to implement and that would not break their extensions. Minor version can also mark method to be changed or removed as deprecated.
• Major versions can make breaking changes such as making optional methods introduced in a previous minor version non-optional and removing deprecated methods as well as changing method signatures.

4. Define Extensibility Conformance rules for extenders

• Issue Document code samples that supports the Zowe Explorer Conformance Criteria #672: We need to provide Conformance rules that would augment the current Zowe Conformance rules with specific rules for a Zowe Explorer extensions. The rules could be based on the specific API implemented by the extension as listed under (1), i.e. each of the groups could define their own conformance rules depending on which were utilized by an extension.

Proposal to provide different sets of criteria for extension for

0. General extension requirements.
1. Profiles access: An extension that accesses the Zowe Explorer Zowe CLI profiles caches for read or write operations.
2. Data Provider: An extension that provides an alternative Zowe CLI profile type for a different z/OS communication protocol for one or more Zowe Explorer views.
3. Menus: An extension that injects new menus into the existing Zowe Explorer tree views accessing contextual information from the nodes.

An extender can then claim for conformance to one or many of these four. For the 1.x releases I think we can only provide conformance rules for (3). For 2.x we should carefully design the extensibility and criteria for the other three.

See this Comment below for a proposal for how the compliance criteria could be defined for the public.

Initial Groundwork

• Issue Migrate from npm to Yarn #905 Migrate from npm to Yarn (phaumer)
• Issue Set up an example package with ESlint plug-in #908 Set up an example package with ESLint plug-in (phaumer)
• Issue Create a package for the API #906 Create a package for the API (phaumer)
• Issue Migrate from from deprecated test runner for integrations tests #1049 Migrate from deprecated integration test runner (Lauren)
• Issue Create shell script to publish scripts out of the Jenkinsfile #907 Create shell script to publish scripts out of the Jenkins file (Fernando)

Next Steps for Q1/2021

• Move more functionality into the API, such as initializing secure credentials store, merging of base profiles
• Documenting the API and providing examples
• Extending the FTP extension to support MVS, JES
• Reviewing the current API and proposing changes to address open issues
• Proposing internal governance rules and verification scripts
• Proposing external compliance rules

Next Steps for Q2/2021

• Issue Add profile functionalities for Zowe Explorer FTP extension #1284 Add profile functionalities for Zowe Explorer FTP extension
• Issue Test automation for FTP package #1028 Test automation for FTP package

[zdmullen]

After a discussion, we've decided to at least prepare with Yarn.

Any dependency scans that are run for Zowe Explorer, they would have to be extended to support Yarn lock files.

Currently the Jenkins file depends on the project being npm. We'll have to use more generic libraries that the Theia pipeline is using.

[zdmullen]

Extract the publishing script out of the Jenkins file into its own file.

*Dan is working to convert also the Theia pipeline to a GitHub Action. A PR will be (re-)open(ed) eventually.

[phaumer]

Link to Zowe project's conformance criteria sheet: https://ent.box.com/s/hqzfsia30z09hsjo8gj40kunhjaw9ebj

[phaumer]

Proposal for Zowe Explorer Conformance Criteria for Zowe Conformance v1

Kinds of extensions

The following kinds of extensions can be certified for Zowe Explorer. One extension offering needs to comply with at least one of these kinds, but it can also be in several or all of the kinds listed. In addition to fulfilling all the Required criteria for at least one kind (1 to 3), the candidate extension also needs to fulfill the required criteria of the (0) General category.

0. General extension.
1. Profiles access: An extension that accesses the Zowe Explorer Zowe CLI profiles caches for read or write operations.
2. Data Provider: An extension that provides an alternative Zowe CLI profile type for a different z/OS communication protocol for one or more Zowe Explorer views.
3. Menus: An extension that injects new menus into the existing Zowe Explorer tree views accessing contextual information from the nodes.

Format for conformance criteria

Every conformance criteria needs to be clearly scoped, defined, and objectively verifiable. We therefore specify the following in each of the criteria descriptions below:

• Unique name: each criteria has a short unique name that can be used for reference and discussion amongst parties
• Required vs Best Practice: we currently distinguish if a criteria is a must-do for conformance versus if it is just a recommended best practice.
• Goal: each criteria should in its description clearly state the goal or rationale for requiring it.
• Testability: Each criteria should specify the way and extender as well as the Zowe Explorer squad can verify conformance.

General conformance criteria (0)

These criteria are independent of the kind of extension produce and need to be fulfill for all cases.

1. (Required) Naming: If the extension uses the word "Zowe" in its name, it abides by Linux Foundation's Trademark Usage rules to ensure the word Zowe is used in a way intended by the Zowe community. The extender needs to provide a link where the offering will be announced and/or made available to the public.
2. (Required): No Zowe CLI plugin installation requirement: If the extender makes use of a Zowe CLI profile other than the Zowe Explorer default zosmf then the extension must not make any assumptions that a matching Zowe CLI plugin has been installed in the Zowe Explorer user's environment. In other words the extension must be full self-contained including all the code of the Zowe CLI Plugin that implements the new profile. This will not only simplify the end user experience, but also ensures that the extension can run in other VS Code compatible environments such as Cloud IDEs such as Eclipse Che. For VS Code extensions Zowe Explorer provides APIs to call that ensure that the users can store and access such new profiles in Zowe home directory folders and secure credentials store, which should be used or an equivalent needs to be provided. To test this requirement a user shall be able start the extension and use it without having Nodejs and Zowe CLI installed locally.
3. (Required) Publication tag: If the extension is published in a public catalog or marketplace such as Npmjs, Open-VSX, or VS Code Marketplace then it must use the tag or keyword "Zowe" so it can be found when searching for Zowe and be listed with other Zowe offerings. For example, if the extension is a VS Code extension then the "keywords" array of the package.json must include the entry "Zowe". This can be verified by looking at the page's Tags section in the VS Code Marketplace or if not yet present by extracting the vsix file and inspecting the package.json file.
4. (Required) Support: The extension needs to document how a user would get support or file issues for the extension for problems that are not related to Zowe and Zowe Explorer. The extender needs to document where the user can find this information in the conformance application.
5. (Best Practice) User settings consistency: For a consistent user experience we recommend that user settings and configuration settings follow the naming conventions as documented in the Zowe Explorer extensibility documentation. The list and documentation of the configuration settings should be made available as part of the compliance application. If the extension is a VS Code extensions then including the package.json file will be sufficient unless other means of configuration settings (such as environment variables) have been used.
6. (Best Practice) Error message consistency: For a consistent user experience we recommend that error and log messages follow the format documented in the Zowe Explorer extensibility documentation. To verify this requirement it is sufficient to provide links to user documentation showing screen shots of error dialogs as well as sample log file entries.
7. (Best Practice) Zowe SDK usage: It is recommended that if possible the the available Zowe SDKs should be utilized to standardize on z/OS interactions as well as other common capabilities used by many other Zowe extensions and tools. This can be verified by reviewing the dependencies list of the implementation such as in the package.json file of a nodejs-based extension.

Profiles Access extension conformance criteria (1)

A Profiles Access extension is a Zowe Explorer extension that uses the Zowe Extensibility API to conveniently access Zowe CLI profiles loaded by Zowe Explorer itself. This allows the extension to consistently access profile instances of specific types, offer them for edit and updates as well as common refresh operations that apply to all extensions, add more profile types it is using itself for its own custom views (for example a CICS extension adding a CICS explorer view) and other similar use cases related to Zowe CLI profiles. These extensions do not have to be VS Code extension if it just wants to use ProfilesCache implementation of Zowe Explorer as all APIs are provided free of any VS Code dependencies. Such an extension could be used for another non-VS Code tool, a Zowe CLI plugin, a Web Server or another technology. However, to access the profiles cache of the actual running VS Code Zowe Explorer the extender needs to be a VS Code extension that has an extension dependency defined to be able to query the extender APIs. Therefore, some of the criteria that are listed here as required are only required if the extender is a VS Code extension.

1. (Required if VS Code) VS Code extension dependency: Zowe Explorer VS Code extensions can only be activated after Zowe Explorer is fully activated itself. Therefore, to ensure the correct activation order and extender must include the following entry in their package.json file. The package.json file must be provided to verify this criteria.

   "extensionDependencies": [
    "Zowe.vscode-extension-for-zowe"
   ],

2. (Required if VS Code) Zowe Extender access: Zowe Explorer VS Code extensions must access the shared Zowe Explorer profiles cache only via the ZoweExplorerApi.IApiRegisterClient.getExplorerExtenderApi() API as documented in the Zowe Explorer extensibility documentation. API calls to the Explorer Extender API will be logged by Zowe Explorer and be verified by inspecting Zowe Explorer log files.
3. (Required if VS Code) Added Profile Type initialization: If the extender depends on a new Zowe CLI profile type other than the Zowe Explorer default zosmf then an extender needs to call ZoweExplorerApi.IApiRegisterClient.getExplorerExtenderApi().initialize(profileTypeName) (TBD) as its first API call to ensure that the profile type can be supported. This call ensures that the profiles of this type can be successfully managed even without the Zowe CLI plugin being installed locally for pure VS Code users as well as that the profiles can successfully be accessed in the secure credentials store. This registration call will be logged by Zowe Explorer and be verified by inspecting Zowe Explorer log files.

Data Provider extension conformance criteria (2)

A data provider Zowe Explorer extension provides an alternative protocol for Zowe Explorer to interact with z/OS. The default protocol Zowe Explorer uses is the z/OSMF REST APIs and data provider adds support for another API. For example, the Zowe Explorer extension for zFTP, which is maintained by the Zowe Explorer squad is an example for a Zowe Explorer data provider extension that uses FTP instead of z/OSMF for all of its USS and MVS interactions. To achieve such an extension it uses a Zowe CLI Plugin for FTP that implemented the core interactions and provided them as an SDK andd

1. (Required) A new data provider requires a new Zowe CLI profile type as it is the foundation for registering the APIs that add the new protocol as documented in the extension guideline. When new APIs get registered via the ZoweExplorerApi.IApiRegisterClient.registerMvsApi() call the APIs passed as a parameter needs to use a unique profile type name. Such registration events will be logged by Zowe Explorer and can be used to verify to successful implementation of the requirement in the Zowe Explorer logs.
2. (Best Practice) Matching Zowe CLI Plugin: It is a best practice to provide a Zowe CLI Plugin for the data provider's profile type that implements the core capabilities requires for the new protocol that users can then also use to interact with the protocol outside of the Zowe Explorer extension using Zowe CLI commands. A Zowe CLI Plugin is a regular NPM package that then can be completely imported as a dependency into a VS Code extension that represents a data provider for Zowe Explorer. The zFTP Zowe CLI Plugin and Zowe Explorer extension for zFTP are an example for such a pairing. This requirement can be verified by testing the CLI plugin and seeing it being used in the package.json file of the VS Code extension.
3. (Required) Data provider API implementation: The Zowe Explorer extensibility API provides interfaces for the required operations to be implemented for MVS, JESS and USS. An extender must either fully implement and register at least one of these three interfaces or alternatively throw JavaScript exceptions that provide meaningful error messages to the end-user in the Error.message property of the exception that Zowe Explorer will display in a dialog. "Not yet implemented by the XYZ data provider" could be an example. Registering a new data provider API instance will be logged by Zowe Explorer and be be verified in the Zowe Explorer log files.
4. (Best practice) If the extension implements a Zowe Explorer API data provider interface, it should implement a test suite that calls each of the implemented API methods. Test run logs can be provided to demonstrate the fulfillment of the requirement.

Menu extension conformance criteria (3)

1. (Required) Menu Names: If the extension is adding new commands and context menu entries to the Zowe Explorer tree view nodes, it adheres to the Zowe Explorer-provided contextual string format. TBD: Define rules and verification criteria.

For team discussion: We currently only have documented for how to add context menus, but not how the commands that implement the menu can actually access the node on which the click occurred to determine all the information needed (e.g. the data set member name and the profile of the node). We need to implement and provide an API for our tree browsers to properly support this and prevent users for just hacking the tree views themselves. Until we have such an API perhaps this extensibility kind needs to be postponed.

[phaumer]

Created a PR to manage the comment above under source control with PRs: #1290

[zFernand0]

Closing.
See #1791 for remaining work on this.