Skip to content
This repository has been archived by the owner on Nov 13, 2023. It is now read-only.

Design Proposal for Zowe Profile Information API #556

Closed
gejohnston opened this issue Mar 1, 2021 · 37 comments
Closed

Design Proposal for Zowe Profile Information API #556

gejohnston opened this issue Mar 1, 2021 · 37 comments
Assignees
Labels
enhancement New feature or request team-profile

Comments

@gejohnston
Copy link
Member

API to access Zowe Profile Information

The Zowe CLI will deprecate its older style profiles for configuring the Zowe CLI, Zowe plugins, and Zowe VS Code extensions. Eventually the old style profiles will be removed from the product. Those old profiles will be replaced by a team configuration file (also referred to as single configuration file, project configuration, and Zowe V2 configuration).

There are several variations to provide flexibility, but in its simplest form, a team configuration is a file named "zowe.config.json" in a user’s ZOWE home directory. The config file contains connection arguments and profile arguments for the Zowe command groups, including every installed plugin. A few fundamental actions are performed on the team configuration with the “zowe config” command group. Users will perform most of their modifications to the configuration by editing zowe.config.json in a text editor.

One must understand the purpose of a team configuration before being able to fully appreciate the functionality of an API that accesses the team configuration. An introduction to the functionality and use of a team configuration can be found in "Zowe CLI — Getting Started, Made Easy!".

While some APIs existed for old style profiles, the Zowe CLI commands, plugins, and VS Code extensions pieced together those APIs differently, or accessed the profiles directly. This led to inconsistent interpretation of the profile information. As new capabilities were written in one component, other components did not demonstrate that new functionality.

During the same time-frame that the new team configuration will be rolled out, we also plan to deliver an API to provide profile information to Zowe components. This document describes the intended functionality of the ProfileInfo API.

Requirements

  1. Ability to read the configuration as stored by the user.

  2. Team configurations have multiple layers. Resolve the value overrides from those layers into a final set of values.

  3. Provide an indicator for a team config, specifying whether the profile is global or project-based.

  4. When searching for a project team config, a caller should be able to specify the starting directory (instead of only searching from the current directory).

  5. Provide a list of all typed profiles.

    • Provide the default profile choice for each type of profile.

    • Optionally limit the list to a specific type of profile.

  6. Retrieve the name of the default profile for a given type of profile.

  7. Apps have the need to resolve the arguments for a particular type of profile. They use this to display a set of choices to the end user.

  8. During the period of deprecation of old style profiles, automatically detect whether team config or old profiles are in use.

    • Automatically read the right style (old profiles or team config) for the API consumer.

    • Provide an indicator of whether old profiles or team config is being used.

    • Provide file paths to config files so they can be launched for editing.

      • On Windows the paths should be treated in a case-insensitive fashion. On Linux they must be case sensitive.
    • One existing function that should be augmented is getDefaultProfile( ).

      • This method loads the default old-style profile of type X.

      • It also looks for a default base profile, and if it exists then merges values between base and service profiles.

      • An enhanced version of getDefaultProfile should also handle team config.

        • If zowe.config.json does not exist, fall back to current implementation for loading old-style profiles

        • If zowe.config.json does exist, load the default service and base profile out of the team config JSON file(s)

  9. Automatically resolve the order of precedence overrides of argument values.

    • A profile’s argument values are set with this precedence:

      • Command line options or GUI input directly from a user.

        • User-supplied options in a CLI are known up-front.

        • User-supplied options in a GUI would typically be acquired after the values merged from all of the other sources below have been shown to the user, and then provide the ability to add to (or override) the existing values.

      • Environment variables.

      • Values stored in a service profile within a Zowe configuration on disk (old profiles or team config).

      • Values stored in a base profile within a Zowe configuration on disk (old profiles or team config).

      • Default values specified in a profile’s definition.

    • Perhaps accept custom args and env objects as input, with default values of process.argv and process.env

    • Need the ability to reload the configuration from disk and re-resolve the order of precedence on demand.

  10. Ensure that the "zowe auth login" action stores the resulting token in the correct profile.

  11. Credential management is strongly related to the management of configuration arguments. Supply a method that appropriately handles the loading of keytar.

    • If zowe.config.json does not exist, it would conditionally load CredentialManager based on the properties in "settings.json"

      • Check in ~/.zowe/imperative/settings.json if CredentialManager is enabled

      • Override keytar module on the DefaultCredentialManager class with VS Code's bundled Keytar

      • Initialize CredentialManagerFactory class to load/save secure credentials with the service name "Zowe-Plugin"

    • If zowe.config.json does exist, it would always load CredentialManager

      • Accepts an optional callback to require a custom Keytar module

      • Returns boolean to indicate whether secure credential manager initialized successfully (or maybe it should throw error if initialization fails?)

Use case considerations

  1. Considerations for GUI config vs editing the config in an editor:

    • Saving user values back to disk is likely the most difficult operation to support in the API and implement in the GUI.

    • How does a GUI save a config value if the user overrode a value that was originally obtained from an environment variable or from the default value within a profile definition?

    • The flexibility of team config is greater than old profiles. It seems problematic to represent the full functionality of both very different sets of capabilities in a single representation.

      • A single representation would increase the complexity of the data structure needed for a GUI to store values back to disk.

      • Returning two different representations puts the burden on the API consumer to implement two sets of processing logic for the two representations.

    • There has been some discussion that a GUI for accepting configuration values from a user might be limited to the most simple case configuration.

      • Do we need to allow a GUI to read and save all configurations in all of their complexities?

      • Should a GUI create a configuration only from scratch, and use a very simple structure?

      • Should we limit GUI editing to only configurations that were created by that GUI?

      • Should a GUI place the user into an editor window to directly edit the config file?

  2. Current imperative functions automatically prompt (on the console) for missing connection arguments. An API should not do console prompting. Possible solutions include:

    • Structure the API so that the caller can retrieve a list of all missing connection arguments so that the caller can prompt for those arguments. The caller then supplies all of the arguments provided by the user in a request to perform an action.

    • Use callbacks to the caller when a connection argument is missing.

      • A shortcoming of callbacks is that a user of a GUI would be asked for each argument one at a time.
  3. In addition to known connection properties, we can provide a list of any required profile arguments that do not have values assigned, so that the caller can prompt for all missing arguments.

    • We could use the "required" property of an argument to add any required, but missing profile arguments to the list of missing arguments.

    • If we use this design, we may be able to restore the "required" property to our set of known connection arguments (like host, port, etc).

    • Note that additional arguments are often required for a command, which are not part of a profile. For example, a "list dataset" command requires a “datasetName” argument. However, datasetName is not an argument defined in a zosmf profile.

    • The ProfileInfo API is limited to arguments that have been defined for a profile itself.

  4. Since our current business direction is to have users edit the team config files, the ProfileInfo API will not support writing values to a Team configuration. The lower-level Config class will be the only means for saving values to the Team configuration.

Similarly the ProfileInfo API will not save values to old-style profiles. Existing (but deprecated) functions must be used to store values in old-style profiles. If callers of the ProfileInfo API need to save to both old profiles and new team configurations, they must write dual-path logic to handle both conditions.

Note that both the old style profile CLI commands to set profiles, and the new "config set" CLI commands will be available.

API functions

The ProfileInfo functions provide the following functionality:

  • Read configuration from disk.

  • Transparently read either new team configuration or old style profiles.

  • Resolve order of precedence for profile argument values.

  • Provide information to enable callers to prompt for missing, but required profile arguments.

The following detailed API documentation was generated from source files which reside in the "profInfo-for-next" branch of the imperative repository (https://github.com/zowe/imperative).

I think that the best way to read the ProfileInfo API is to read the "Pseudocode examples" at the beginning of the ProfileInfo API doc, and then look for the details about any function that interests you.

The API details are listed below in this order:

Recommended ProfileInfo API

This class provides functions to retrieve profile-related information.
It can load the relevant configuration files, merge all possible profile
argument values using the Zowe order-of-precedence, and access desired
profile attributes from the Zowe configuration settings.

Pseudocode examples:

   // Construct a new object. Use it to read the profiles from disk
   profInfo = new ProfileInfo();
   profInfo.readProfilesFromDisk("zowe");

   // Maybe you want the list of all zosmf profiles
   let arrayOfProfiles = profInfo.getAllProfiles("zosmf");
   youDisplayTheListOfProfiles(arrayOfProfiles);

   // Maybe you want the default zosmf profile
   let zosmfProfile = profInfo.getDefaultProfile("zosmf");
   youUseTheProfile(zosmfProfile);

   // Maybe you want the arg values for the default JCLCheck profile
   let jckProfile = profInfo.getDefaultProfile("jclcheck");
   let jckMergedArgs = profInfo.mergeArgsForProfile(jckProfile);
   let jckFinalArgs = youPromptForMissingArgsAndCombineWithKnownArgs(
       jckMergedArgs.knownArgs, jckMergedArgs.missingArgs
   );
   youRunJclCheck(jckFinalArgs);

   // Maybe no profile of type "zosmf" even exists.
   let zosmfProfiles = profInfo.getAllProfiles("zosmf");
   if (zosmfProfiles.length == 0) {
       // No zosmf profile exists
       // Merge any required arg values for the zosmf profile type
       let zosmfMergedArgs =
           profInfo.mergeArgsForProfileType("zosmf");

       let finalZosmfArgs =
           youPromptForMissingArgsAndCombineWithKnownArgs(
               zosmfMergedArgs.knownArgs,
               zosmfMergedArgs.missingArgs
           );
       youRunSomeZosmfCommand(finalZosmfArgs);
   }

   // So you want to write to a config file? You must use your own
   // old-school techniques to write to old-school profiles.
   // You then use alternate logic for a team config.
   // You must use the Config API to write to a team configuration.
   // See the Config class documentation for functions to set
   // and save team config arguments.

   // Let's save some zosmf arguments from the example above.
   let yourZosmfArgsToWrite: IProfArgAttrs =
       youSetValuesToOverwrite(
           zosmfMergedArgs.knownArgs, zosmfMergedArgs.missingArgs
       );
   if (profInfo.usingTeamConfig()) {
       let configObj: Config = profInfo.getTeamConfig();
       youWriteArgValuesUsingConfigObj(
           configObj, yourZosmfArgsToWrite
       );
   } else {
     youWriteOldSchoolProfiles(yourZosmfArgsToWrite);
   }

Hierarchy

  • ProfileInfo

Index

Properties

  • mLoadedConfig
  • mUsingTeamConfig

Methods

  • ensureReadFromDisk
  • getAllProfiles
  • getDefaultProfile
  • getTeamConfig
  • mergeArgsForProfile
  • mergeArgsForProfileType
  • readProfilesFromDisk
  • usingTeamConfig

Properties

Private mLoadedConfig

mLoadedConfig:
Config
= null

Private mUsingTeamConfig

mUsingTeamConfig: boolean = false

Methods

Private ensureReadFromDisk

  • ensureReadFromDisk(): void

  • Ensures that ProfileInfo.readProfilesFromDisk() is called before an
    operation that requires that information.

    Returns void

getAllProfiles

  • getAllProfiles(profileType?: string):
    IProfAttrs[]

  • Get all of the typed profiles in the configuration.

    Parameters

    • Optional profileType: string
         Limit selection to only profiles of the specified type.
         If not supplied, the names of all typed profiles are returned.
      

    Returns IProfAttrs[]

    An array of profile attribute objects. In addition to the name, you
    get the profile type, an indicator of whether the profile is the
    default profile for that type, and the location of that profile.

         If no profile exists for the specified type (or if
         no profiles of any kind exist), we return an empty array
         ie, length is zero.
    

getDefaultProfile

  • getDefaultProfile(profileType: string):
    IProfAttrs

  • Get the default profile for the specified profile type.

    Parameters

    • profileType: string
         The type of profile of interest.
      

    Returns IProfAttrs

    The default profile. If no profile exists for the specified type, we
    return null;

getTeamConfig

  • getTeamConfig():
    Config

  • Get the Config object used to manipulate the team configuration on
    disk.

    Our current market direction is to encourage customers to edit the
    team configuration files in their favorite text editor.

    If you must ignore this recommended practice, you must use the
    Config class to manipulate the team config files. This class has a
    more detailed and therefore more complicated API, but it does
    contain functions to write data to the team configuration files.

    You must call ProfileInfo.readProfilesFromDisk() before calling this
    function.

    Returns Config

    An instance of the Config class that can be used to manipulate the
    team configuration on disk.

mergeArgsForProfile

  • mergeArgsForProfile(profile:
    IProfAttrs):
    IProfMergedArg

  • Merge all of the available values for arguments defined for the
    specified profile. Values are retrieved from the following sources.
    Each successive source will override the previous source.

    • A default value for the argument that is defined in the profile
      definition.
    • An environment variable for that argument.
    • A value defined in the base profile.
    • A value defined in the specified service profile.
    • For a team configuration, both the base profile values and the
      service profile values will be overridden with values from a
      zowe.config.user.json file (if it exists).

    Parameters

    • profile: IProfAttrs
         The profile whose arguments are to be merged.
      

    Returns IProfMergedArg

    An object that contains an array of known profile argument values
    and an array of required profile arguments which have no value
    assigned. Either of the two arrays could be of zero length,
    depending on the user's configuration and environment.

         We will return null if the profile does not exist
         in the current Zowe configuration.
    

mergeArgsForProfileType

  • mergeArgsForProfileType(profileType: string):
    IProfMergedArg

  • Merge all of the available values for arguments defined for the
    specified profile type. See mergeArgsForProfile() for details about
    the merging algorithm. The intended use is when no profile of a
    specific type exists. The consumer app can prompt for values for
    missing arguments and then perform the desired operation.

    Parameters

    • profileType: string
         The type of profile of interest.
      

    Returns IProfMergedArg

    The complete set of required properties;

readProfilesFromDisk

  • readProfilesFromDisk(appName: string, teamCfgOpts?:
    IConfigOpts):
    Promise<void>

  • Read either the new team configuration files (if any exist) or read
    the old-school profile files.

    todo: Does our consumer need to call this function for old-school
    profiles?

    Parameters

    • appName: string
         The name of the application (like "zowe" in zowe.config.json)
         whose configuration we want to read.
      
    • Optional teamCfgOpts: IConfigOpts
         The optional choices used when reading a team configuration.
         This parameter is ignored, if the end-user is using old-school
         profiles.
         todo: We must add a startingProjectSearchDir to IConfigOpts.
      

    Returns Promise<void>

usingTeamConfig

  • usingTeamConfig(): boolean

  • Returns an indicator of whether we are using a team configuration or
    old-school profiles.

    You must call ProfileInfo.readProfilesFromDisk() before calling this
    function.

    Returns boolean

    True when we are using a team config. False means old-school
    profiles.

Data structures used by ProfileInfo

The following data structures describe the content of various objects used by the ProfileInfo API.

// _______________________________________________________________________
/**
 * The attributes of a profile argument.
 */
export interface IProfArgAttrs {
    /** The name of the argument */
    argName: string;

    /** The type of data for this property */
    dataType: "string" | "number";

    /** The value for the argument */
    argValue: string | number;

    /** The location of this argument */
    argLoc: IProfLoc;
}

// _______________________________________________________________________
/**
 * The identifying attributes of a profile.
 */
export interface IProfAttrs {
    /** The name of the profile */
    profName: string;

    /** The profile type (eg. "zosmf") */
    profType: string;

    /** Indicates if this is the default profile for this type */
    isDefaultProfile: boolean;

    /**
     * Location of this profile.
     * profNmLoc.ProfLocType can never be ProfLocType.ENV or
     * ProfLocType.DEFAULT, because this is the location of a profile,
     * not an argument value.
     */
    profLoc: IProfLoc;
}

// _______________________________________________________________________
/**
 * This enum represents the type of location for a property.
 * Note that properties with location types of ENV and DEFAULT
 * cannot be stored back to disk. Thus the consumer app must
 * make its own decision about where to store the property.
 */
export enum ProfLocType {
    OLD_PROFILE = 0,    // an old-school profile
    TEAM_CONFIG,        // a team configuration
    ENV,                // an environment variable
    DEFAULT             // the default value from a profile definition
};

/** The attributes used to identify the location of a given property */
export interface IProfLoc {
    /**
     * The type of location for this property
     */
    locType:    ProfLocType.OLD_PROFILE |
                ProfLocType.TEAM_CONFIG |
                ProfLocType.ENV |
                ProfLocType.DEFAULT;

    /**
     * For OLD_PROFILE and TEAM_CONFIG, this is the path to
     * the file on disk which contains the argument.
     * For ENV, this is the name of the environment variable.
     * This is not used for DEFAULT.
     */
    osLoc?: string;

    /**
     * For SOURCE_TEAM_CONFIG, this is the dotted path into
     * the JSON configuration object for the profile.
     * This property is not used for SOURCE_OLD_PROFILE, because
     * the old-shool profiles use a simple ArgumentName: value pair.
     * This property is also not used for ENV or DEFAULT.
     */
    jsonLoc?: string;
};

// _______________________________________________________________________
/**
 * The result object from mergeProfileArgs().
 */
export interface IProfMergedArg {
    /**
     * The list of arguments with known values.
     * All of the attributes in IProfPropAttrs will be filled in except
     * when knownArgs[i].propLoc.osLoc (and/or jsonLoc) are not
     * relevant for the type of location (locType).
     */
    knownArgs: IProfArgAttrs[];

    /**
     * The list of required arguments for which no value has been specified.
     * Obviously, the missingArgs[i].propValue attribute will not exist.
     * Note that a generated team configuration template can contain some
     * arguments with an empty string as a value. Such arguments will be
     * contained in this missing list. The other missing arguments will
     * have a missingArgs[i].propLoc, derived from the location of the
     * profile specified to the function mergeProfileArgs().
     */
    missingArgs: IProfArgAttrs[];
}

Low-level Config API

The Config class provides facilities for reading and writing team
configuration files. It is used by Imperative to perform low-level
operations on a team configuration. The intent is that consumer apps
will not typically use the Config class, since end-users are expected
to write team configuration files by directly editing them in an editor
like VSCode.

Hierarchy

  • Config

Index

Constructors

  • constructor

Properties

  • mActive
  • mApp
  • mHome
  • mLayers
  • mSecure
  • mVault
  • opts
  • END_OF_TEAM_CONFIG
  • END_OF_USER_CONFIG

Accessors

  • api
  • appName
  • configName
  • exists
  • layers
  • maskedProperties
  • paths
  • properties
  • schemaName
  • userConfigName

Methods

  • delete
  • findLayer
  • formMainConfigPathNm
  • layerActive
  • layerMerge
  • layerPath
  • layerProfiles
  • save
  • set
  • setSchema
  • empty
  • load
  • search

Constructors

Private constructor

  • new Config(opts?:
    IConfigOpts):
    Config

  • Constructor for Config class. Don't use this directly. Await
    Config.load instead.

    Parameters

    • Optional opts: IConfigOpts

      Options to control how Config class behaves

    Returns Config

Properties

mActive

mActive: object

Currently active layer whose properties will be manipulated

internal
:

Type declaration

  • global: boolean
  • user: boolean

mApp

mApp: string

App name used in config filenames (e.g., my_cli.config.json)

internal
:

mHome

mHome: string

Directory where global config files are located. Defaults to
~/.appName.

internal
:

mLayers

mLayers:
IConfigLayer[]

List to store each of the config layers enumerated in layers enum

internal
:

mSecure

mSecure:
IConfigSecure

Secure properties object stored in credential vault

internal
:

mVault

mVault:
IConfigVault

Vault object with methods for loading and saving secure credentials

internal
:

Optional opts

opts:
IConfigOpts

Options to control how Config class behaves

Static Private END_OF_TEAM_CONFIG

END_OF_TEAM_CONFIG: ".config.json" = ".config.json"

The trailing portion of a shared config file name

Static Private END_OF_USER_CONFIG

END_OF_USER_CONFIG: ".config.user.json" = ".config.user.json"

The trailing portion of a user-specific config file name

Accessors

api

  • get api(): object

  • Access the config API for manipulating profiles, plugins, layers,
    and secure values.

    Returns object

    • layers: ConfigLayers
    • plugins: ConfigPlugins
    • profiles: ConfigProfiles
    • secure: ConfigSecure

appName

configName

exists

layers

  • get layers():
    IConfigLayer[]

  • List of all config layers. Returns a clone to prevent accidental
    edits of the original object.

    Returns IConfigLayer[]

maskedProperties

  • get maskedProperties():
    IConfig

  • The properties object with secure values masked.

    type
    : {IConfig}

    memberof
    : Config

    Returns IConfig

paths

properties

  • get properties():
    IConfig

  • List of properties across all config layers. Returns a clone to
    prevent accidental edits of the orignal object.

    Returns IConfig

schemaName

userConfigName

Methods

delete

  • delete(path: string, opts?: object): void

  • Unset value of a property in the active config layer.

    Parameters

    • path: string

      Property path

    • Optional opts: object

      Include secure: false to preserve property in secure array

      • Optional secure?: boolean

    Returns void

findLayer

  • findLayer(user: boolean, global: boolean):
    IConfigLayer

  • Find the layer with the specified user and global properties.

    internal
    :

    Parameters

    • user: boolean

      True specifies that you want the user layer.

    • global: boolean

      True specifies that you want the layer at the global level.

    Returns IConfigLayer

    The desired layer object. Null if no layer matches.

formMainConfigPathNm

  • formMainConfigPathNm(options: any): string

  • Form the path name of the team config file to display in messages.
    Always return the team name (not the user name). If the a team
    configuration is active, return the full path to the config file.

    Parameters

    • options: any

      a map containing option properties. Currently, the only property
      supported is a boolean named addPath. {addPath: true | false}

    Returns string

    The path (if requested) and file name of the team config file.

layerActive

  • layerActive():
    IConfigLayer

  • Obtain the layer object that is currently active.

    internal
    :

    Returns IConfigLayer

    The active layer object

Private layerMerge

  • layerMerge(maskSecure?: boolean):
    IConfig

  • Merge the properties from multiple layers into a single Config
    object.

    internal
    :

    Parameters

    • Optional maskSecure: boolean

      Indicates whether we should mask off secure properties.

    Returns IConfig

    The resulting Config object

Private layerPath

  • layerPath(layer:
    Layers):
    string

  • Get absolute file path for a config layer. For project config files,
    We search up from our current directory and ignore the Zowe hone
    directory (in case our current directory is under Zowe home.). For
    golbal config files we only retrieve config files from the Zowe home
    directory.

    internal
    :

    Parameters

    • layer: Layers

      Enum value for config layer

    Returns string

layerProfiles

  • layerProfiles(layer:
    IConfigLayer,
    maskSecure?: boolean): object

  • Obtain the profiles object for a specified layer object.

    internal
    :

    Parameters

    • layer: IConfigLayer

      The layer for which we want the profiles.

    • Optional maskSecure: boolean

      If true, we will mask the values of secure properties.

    Returns object

    The resulting profile object

    • [key: string]: IConfigProfile

save

  • save(allLayers?: boolean): Promise<void>

  • Save config files to disk and store secure properties in vault.

    Parameters

    • Optional allLayers: boolean

      Specify false to save only the active config layer

    Returns Promise<void>

set

  • set(path: string, value: any, opts?: object): void

  • Set value of a property in the active config layer. TODO: more
    validation

    Parameters

    • path: string

      Property path

    • value: any

      Property value

    • Optional opts: object

      Include secure: true to store the property securely

      • Optional secure?: boolean

    Returns void

setSchema

  • setSchema(schema: string | object): void

  • Set the $schema value at the top of the config JSONC. Also save the
    schema to disk if an object is provided.

    Parameters

    • schema: string | object

      The URI of JSON schema, or a schema object to use

    Returns void

Static empty

Static load

  • load(app: string, opts?:
    IConfigOpts):
    Promise<Config>

  • Load config files from disk and secure properties from vault.

    Parameters

    • app: string

      App name used in config filenames (e.g., my_cli.config.json)

    • Optional opts: IConfigOpts

      Options to control how Config class behaves

    Returns Promise<Config>

Static search

  • search(file: string, opts?: object): string

  • Search for up the directory tree for the directory containing the
    specified config file.

    Parameters

    • file: string

      Contains the name of the desired config file

    • Optional opts: object
      • Optional ignoreDirs?: string[]

        Contains an array of direcory names to be ignored (skipped)
        during the search.

    Returns string

    The full path name to config file or null if not found.

@VitGottwald
Copy link

This is a long post. What is the preferred way to provide feedback so that it can stay organised and not get lost in the sheer volume of text? Should I add comments here, open new issues and link here or do something else?

@gejohnston
Copy link
Member Author

This design document was created as a Git Issue instead of a Google Doc to enable squad members who do not work at Broadcom to be able to access the design document.

Even though there is a lot of text, I suggest that we add comments to this issue, so that all ideas can be found in one place.

@VitGottwald
Copy link

VitGottwald commented Mar 9, 2021

Since our current business direction is to have users edit the team config files, the ProfileInfo API will not support writing values to a Team configuration.

I would like to ask for a bit more rationale behind this because it is a breaking change for existing and future functionality of our VSCode extensions.

Existing functionality

  • ZOWE Explorer creates and updates z/OSMF profiles.
  • Endevor Explorer creates endevor profiles, a new version (currently in development) also creates endevor-location profiles.
  • The extenders API has the ability to save and delete profiles.

In development functionality

  • Dataset templates by @katelynienaber are created in VSCode through user actions in the GUI, they are currently stored in vscode settings, but if we would like to reuse them in a CLI plugin for example, it would make sense to write them to the zowe config.

Going forward

  • I expect more requests to be able to write to or update the zowe config as our VSCode extensions become more advanced and want to offer a better UX as well as support for project level configuration of their own.

Environment variables

Unlike in a CLI our VSCode extensions do not use nor have the desire to use environment variables for profile properties. Because of that I do not see

How does a GUI save a config value if the user overrode a value that was originally obtained from an environment variable

as an issue.

@t1m0thyj t1m0thyj added this to the Zowe vNext Backlog milestone Mar 9, 2021
@venkatzhub
Copy link

I agree with @VitGottwald. I also had discussion with @dkelosky on this topic. The "read" API is complex in the sense that it is going to merge and combine the config and provide the callers a comprehensive/merged config file. The same thing need not be true for write or update. It is fair to have more granular write API(s) to handle the various cases.

@gejohnston
Copy link
Member Author

I have some questions for VSCode extension implementers. Your responses can affect how we implement some items.

  • Do you expect that the Zowe CLI is installed on the machine, or do you bundle, import, and use only SDK functions?
  • I believe that you do NOT call Imperative.init(). Is that correct?
    • How have you replicated information that is established in Imperative.init?
      • One example is logging.
        • Do your apps even care about logging?
        • Do you just let default logging occur?
        • Did you add your own logic to collect logging levels from Zowe environment variables?
    • How have you identified arguments that are required for a profile?
      • Do you read xxx_meta.yaml?
      • Do you search for profile definitions within a command's definition, which is typically specified in a command group's imperative.js file?
    • Do you use an argument's data types for validation purposes, or just use the argument names?
    • Are there other Imperative.init operations that you replicate in your own app?
  • Whenever you request that profile data be read from disk, we plan to read all relevant profile data and store it in memory. Other requests for information will come from that in-memory cache, until you once again call the function to read everything from disk.
    • Does that accomplish the goal discussed in our recent Zoom meeting?

@jellypuno
Copy link
Contributor

Do you expect that the Zowe CLI is installed on the machine, or do you bundle, import, and use only SDK functions?

no. not really. we are importing it and use the SDK functions.

I believe that you do NOT call Imperative.init(). Is that correct?

No. as far as I know, we are importing Logger from imperative and this writes a log in .vscode > Zowe Explorer > logs

Did you add your own logic to collect logging levels from Zowe environment variables?

I don't think we are using any Zowe environment variables

Do you read xxx_meta.yaml?

We use this to search for the schema

    public getSchema(profileType: string): Record<string, unknown> {
        const profileManager = this.getCliProfileManager(profileType);
        const configOptions = Array.from(profileManager.configurations);
        let schema = {};
        for (const val of configOptions) {
            if (val.type === profileType) {
                schema = val.schema.properties;
            }
        }
        return schema;
    }

Do you use an argument's data types for validation purposes, or just use the argument names?
Are there other Imperative.init operations that you replicate in your own app?

I'm not sure.

Whenever you request that profile data be read from disk, we plan to read all relevant profile data and store it in memory. Other requests for information will come from that in-memory cache, until you once again call the function to read everything from disk.
Does that accomplish the goal discussed in our recent Zoom meeting?

It would be great to share that 'profile data stored in memory'. Currently we are storing it ourselves and this is probably the same for other extensions. So if we can access the memory from CLI then that would be easier for all of us

@venkatzhub
Copy link

Thanks @jellypuno. The key point here is, we need to take into consideration all the other extensions in play here as well - and not just Zowe Explorer.

@awharn
Copy link
Member

awharn commented Mar 15, 2021

I've started implementation for getDefaultProfile, and have run into a small quirk with the team config. A "profile" can be sourced from the combination of multiple team configuration files, which is problematic for the IProfLoc interface, as the osLoc property is currently a string and can only be assigned to one of them.

I intend to modify the interface to make osLoc a list of strings. While all old-school profiles and most team profiles will only have one entry in the list, I believe it is important that the value provide a complete picture of where these profiles, and their properties, are coming from. In most cases, using the first value of the list is perfectly acceptable to add properties to a profile. However, if a profile property is already defined, its origin is not necessarily in the first location, and failure to include any alternate locations may lead to confusion in the future.

If there is any feedback or considerations on this proposed change, it would be greatly appreciated.

@jellypuno
Copy link
Contributor

@awharn We are hoping that we could use profLoc when a user wants to update their config file. The possible flow would be:

  1. User right-clicks on the profile and hits update profile
  2. If ZE detects that the user uses a config file, we will check the profLoc var to get the location of the file
  3. We will open that file in the editor.

So based on my understanding on your modifications, profLoc or osLoc may contain multiple config locations. If we want to trigger the update, then it means we would need to open all file location that is specified in the variable?

@awharn
Copy link
Member

awharn commented Mar 15, 2021

If this change is made, osLoc could contain multiple config locations, but only if the profile exists in multiple config files. Most of the time, I would expect the list to contain one element, but there may be cases where there are two or more elements (i.e. a team member entered the host and port in the team file, and a user entered their TSO account number in the user file). In that case, it may be necessary to open all of the file locations specified in the variable. Otherwise, it is an incomplete view of the profile's settings.

@jellypuno
Copy link
Contributor

@awharn ok. that makes sense. I could imagine a scenario where I have a global config where the system's host and port are located. Then I have a user config that contains my credentials. One issue that I'm thinking is that I shouldn't be able to edit my global config but that is another story.

@gejohnston
Copy link
Member Author

@jellypuno Regarding logging:

Our current implementation accepts an "appName" on its constructor. We use the appName to:

  1. Form the name of the team config file like this: appName.config.json. If "zowe" is passed as the appName, this results in zowe.config.json.
  2. Form the name of the logging directory within the ZOWE CLI HOME directory. If not overridden by an an environment variable named 'appName_CLI_HOME', we use the user's OS home directory. For example, my log messages would go to C:\Users\ej608771\.zowe\imperative\logs\imperative.log

This seems quite different from (what I assume is a VS Code convention of) .vscode > Zowe Explorer > logs.

Would it be preferable to allow the consuming app to pass an Imperative Logger object as another, optional argument on our constructor? If supplied, we would just use that logger object instead of creating one with the characteristics that I described above.

@VitGottwald
Copy link

In Endevor Explorer and Bridge For Git explorer we only import CliProfileManager from imperative and use it to read and write profiles. We have a boundary module that encapsulates all we need from profiles, merges base and service profile data and performs data validation to return properly typed values.

I put it here so that you can look at it.

@VitGottwald
Copy link

VitGottwald commented Mar 15, 2021

As far as logging goes, I think it would be more natural to use VSCode output channel like we do for the Endevor extensions' logging here . Having the ability to supply our own logger that would log into vscode channel would be helpful.

We also currently mock a Console module in tests because they were polluted by many imperative logs and we did not find a better way to disable it. Having the ability to provide a mock module directly, would make the tests cleaner.

From Gene on Mar 16:

There are enough separate threads in this issue that I have chosen to edit this comment, solely to add my additional thoughts. Let me know if doing such edits is a problem.

I failed to describe the reason for the logging initialization currently done by the ProfileInfo constructor. When I ran some unit tests, log messages from underlying Imperative functions were displayed in my command window. This was because I had not initialized the Zowe Logger class, and messages went to stdout. Historically, for Windows GUI apps, such stdout messages silently disappear. Either displaying log messages to stdout or completely losing messages would be undesirable behavior. I initialized the logger so that those messages would go to an imperative log file. I think the ProfileInfo constructor could automatically determine if an Imperative Logger has already been initialized, and only perform a Logger initialization if needed.

I can see how VSCode apps would want to write logs in a VSCode-compliant fashion. It seems unrealistic to get every underlying Imperative utility function to adhere to VSCode logging conventions. I suggest that we let VSCode apps log messages from their app in their own VSCode-compliant fashion. Depending on whether Imperative Logging has already been initialized, ProfileInfo would either use the existing Imperative Logger, or initialize a default location. If ProfileInfo initializes logging, it would choose the location as I described in a previous comment.

If you need to call a different Imperative function before ProfileInfo, and want to have ProfileInfo initialize logging for you, you could create a new ProfileInfo object (which will initialize the Imperative logger), call the other Imperative functions, and then later use the ProfileInfo object.

For Example:

profInfo = new ProfileInfo("zowe"); // this will initialize the Imperative logger
// You can now call other Imperative functions and later use ProfileInfo functions
SomeOtherImperativeFunction();
profInfo.getDefaultProfile("zosmf");

@VitGottwald
Copy link

VitGottwald commented Mar 15, 2021

Regarding profile cache, I will have to look at the implementation in Zowe Explorer that @phaumer mentioned to better understand how it is used. In Endevor explorer we try to stay as close to the real values on disk as possible and avoid managing our own copy of profiles in memory.

From Gene on Mar 16:

We may be inching into mutually exclusive requirements. We heard from the Zowe Explorer team that it was really important to only read from disk when requested. This current comment implies that direct input from disk for every action is important.

It was our intention that you could control how frequently you read from disk by calling ProfileInfo.readProfilesFromDisk as frequently as you need. Functions that provided various attributes and argument values would deliver that information from what was already read from disk.

@VitGottwald
Copy link

@awharn what is the meaning of IProfAttrs.profLoc.jsonLoc? It seems that the ProfLocType interface is being used for two different things

  1. where a particular attribute of a profile comes from
  2. what are all the places a given profile draws attributes from

would it, perhaps, be better to have two interfaces that each focus on a given concern?

@jellypuno
Copy link
Contributor

jellypuno commented Mar 16, 2021

Thanks @jellypuno. The key point here is, we need to take into consideration all the other extensions in play here as well - and not just Zowe Explorer.

@VitGottwald @venkatzhub @dkelosky Are you planning on implementing the SDK on E4E as well? Does the consideration that you intend here is to look at all the different extensions and mold the architecture of the SDK based on all those extensions? Or should the architecture be focused on Zowe Explorer and then the extenders will make use of that?

@dkelosky
Copy link
Contributor

We talked about doing this for E4E, but I'm not sure if it's explicitly captured in our backlog.

The SDK is intended for use by all clients; not just Zowe Explorer. Eventually we would plan to use this SDK for other VS Code extensions, like our JCL extension and sample / template extensions which are yet to be created.

I don't think all Zowe VS Code extensions would pre-req Zowe Explorer.

On the CLI, we require all plugins to be installed into the CLI, i.e. plugins must not run standalone. However, Zowe Explorer itself is an extension to VS Code; it's like a plugin itself.

Other Zowe-related extensions (like E4E) can provide value even without Zowe Explorer in the picture. Perhaps pre-req'ing Zowe Explorer would be a "best practice" but not a hard requirement.

@VitGottwald
Copy link

Thanks @jellypuno. The key point here is, we need to take into consideration all the other extensions in play here as well - and not just Zowe Explorer.

@VitGottwald @venkatzhub @dkelosky Are you planning on implementing the SDK on E4E as well? Does the consideration that you intend here is to look at all the different extensions and mold the architecture of the SDK based on all those extensions? Or should the architecture be focused on Zowe Explorer and then the extenders will make use of that?

Once the single config is rolled out, there will be a natural pressure to use it in other extensions. Having a strategy where some extensions use the old yaml files and some use the new config does not sound like a good idea long term.

@VitGottwald
Copy link

VitGottwald commented Mar 16, 2021

So we should really take a look a what the impact is for other extensions while it is being worked on. And raise requirements. If we do not do this homework we will be putting ourselves into a difficult position in the future.

@dkelosky
Copy link
Contributor

More than natural pressure; we'll eventually require it for conformance 😄

We'll be testing the SDK with an internal VS Code Extension - perhaps you can try for E4E as well. Once we're fairly confident we have something that is workable for a few types of extensions, we'll again get feedback from the broader community.

@venkatzhub
Copy link

@jellypuno - I agree that we cannot afford to have a mixed bag, and the current direction is for every extension to use the new config. We absolutely have to look at the other extensions (I was under the impression that it was done already, but looks like it was not :( )

@jellypuno
Copy link
Contributor

@venkatzhub It was not part of our plan. The plan (at least for us) is to implement it in Zowe Explorer first then test it in zFTP extension. make sure that it is working. If we are planning to add E4E in the picture then our engineers don't have capacity for that. Maybe other extensions like what @dkelosky said or next PI?

@dkelosky
Copy link
Contributor

@jellypuno - I agree that we cannot afford to have a mixed bag, and the current direction is for every extension to use the new config. We absolutely have to look at the other extensions (I was under the impression that it was done already, but looks like it was not :( )

It depends on what is meant by "look at". We're aware of and track as many Zowe VS Code extensions as we have access to. We have looked into repos ourselves or otherwise contacted those responsible for the repos to share our plans. We're also communicating in the Zowe community about our SDK plans and asked for input from all workstation clients (not just VS Code extensions). We're fairly confident that we (collectively) know our SDK will support early adopters.

If we've missed some team, please direct them here 😄. This is all still in validation phase; we can accommodate changes as we develop. Even after, if there is something we miss, we should be able to accommodate then as well.

We've demonstrated that we can roll out changes to all CLI plugins, and I don't see this as being that different. Of course, we had an advantage there with our conformance criteria to ensure consistency. We don't have this yet for extensions, but we're working on it.

@venkatzhub
Copy link

@jellypuno - I'm fine with the plan of handling ZE first. My comment was more from a design and architecture point of view - where the understanding of the requirements from other extensions like E4E etc. should be taken into account. As @dkelosky said, the teams have been contacted, but what is not clear to me is if the teams got a chance to think thru all the aspects - and if they were able to articulate the requirements. If that is done, we are in good shape, if not - I think we need to evaluate if the teams have time to at least understand what the new config would mean to them.

@dkelosky
Copy link
Contributor

Likely they have not gone through all aspects - this stuff isn't on peoples radar until it's time to use it. Internally - I tagged a few other folks to take another look and provide feedback here. We'll also make a point to mention it on a few sync meetings this week. Thank you

@gejohnston
Copy link
Member Author

@awharn what is the meaning of IProfAttrs.profLoc.jsonLoc? It seems that the ProfLocType interface is being used for two different things

1. where a particular attribute of a profile comes from

2. what are all the places a given profile draws attributes from

would it, perhaps, be better to have two interfaces that each focus on a given concern?

The IProfAttrs.profLoc.jsonLoc property is ONLY used when the item in question comes from a Team Configuration. In a team config, one needs to know where in the JSON document an item exists, for example, "ca11.myDefaultZosmfProfile.port".

We anticipate that profLoc would typically be needed if you choose to write data back to disk.

I originally had profile location and argument location as two separate interfaces. Most of the properties were identical between the two, so much so, that I found that I was frequently confused between the two. That is when I combined them into one interface, with a couple of property values not used for one of the types.

For a profile, only ProfLocType.OLD_PROFILE or ProfLocType.TEAM_CONFIG apply.

For an argument value, any of ProfLocType.OLD_PROFILE, ProfLocType.TEAM_CONFIG, ProfLocType.ENV, or ProfLocType.DEFAULT would apply.

@VitGottwald
Copy link

VitGottwald commented Mar 16, 2021

Likely they have not gone through all aspects - this stuff isn't on peoples radar until it's time to use it. Internally - I tagged a few other folks to take another look and provide feedback here. We'll also make a point to mention it on a few sync meetings this week. Thank you

As the devil is usually in the details, I think it would be good to make sure, that whoever looks a this understands the inheritance structure in service profiles and the complexity it adds to the solution - making updates through API very difficult to perform.

I think this is an aspect that is not really emphasised in presentations.

@gejohnston
Copy link
Member Author

@jellypuno & @VitGottwald

To keep some of my responses associated with your original comments, I edited some of those comments and added my responses.

However, it appears that no notifications are sent by doing an edit, so I will not do that anymore.

To find those replies, you can search this issue for the following text:

"From Gene on Mar 16"

@ishche
Copy link

ishche commented Mar 18, 2021

As I remember, in order to access secure password storage, a user may be asked to provide an access password.
Should this API cover this interaction?

@t1m0thyj
Copy link
Member

@ishche Are you referring to this kind of prompt that Zowe CLI/Explorer shows on macOS when accessing passwords?
image

If so, this is OS behavior that we don't have control over. macOS automatically prompts for the keychain password whenever Zowe CLI attempts to load secure passwords.

Although we're not able to directly cover this interaction with an API, we have taken steps to reduce the number of prompts. If you're using the new team config, you should only be prompted for a keychain password once, since secure credentials for all profiles are loaded all at once from a single vault entry. This hopefully provides a better UX on macOS than the old profiles, which prompted for a password many times for each secure credential being accessed.

@gejohnston
Copy link
Member Author

We are interested in your opinions about calling Imperative.init.

We understand that your VSCode apps do not currently call Imperative.init. In fact, we believe that some of you have removed a call to that function from your app. We are interested in hearing whether you have serious objections if the ProfileInfo API were to automatically call Imperative.init for you. The following are the trade-offs that we are considering. We would like to know if you have strong objections to either of the following approaches for gathering profile definitions.

The problem to be solved

Profiles themselves contain argument names and values. Information about what arguments are required, their data types, and their default values are contained within profile definitions. The ProfileInfo API must obtain those profile definitions.

Read profile definitions from disk files

We can read the profile definitions from disk files (the schema file for Team-config, and XXX_meta.yaml files for old profiles).

  • Pros

    • We would not have the performance overhead of calling Imperative.init.
      • Have you noticed significant delays when you called Imperative.init from your VSCode apps in the past?
    • We would avoid a possible reliance on the Zowe-CLI being installed for a VSCode app to run successfully.
  • Cons

    • Even though Zowe-CLI commands will run ok when the schema does not exist, the ProfileInfo API cannot supply a list of missing arguments to VSCode apps.
      • Our recommended practice would be to copy both zowe.config.json and zowe.schema.json to any user trying to get started.
      • When a schema file does not exist, the ProfileInfo API would probably have to return some sort of error code indicating that we can only get values supplied in the config file, but we cannot determine missing arguments.
    • Technically, the same would be true if old-style meta.yaml files were missing.
      • For old-style profiles, it is less likely that just the meta files would be missing. It is more likely that the entire profiles directory is missing, in which case, we would also be unable to collect known values.
    • A schema file can be identified by a file name, but also by a URL.
      • A URL would introduce the need to perform a REST call to obtain the schema definition, which would introduce added complexity and the need to react to additional errors (like "page not found", "access denied", etc).
    • If we only read data from an existing schema file we will be unable to add an API function in the future to perform an operation like "zowe config init". The purpose of such an operation would be to create a schema file and a template team config file.
      • A "config init" operation needs the profile definitions to be able to create the schema. We would not have access to the profile definitions because we never called the Imperative.init function.

Read profile definitions through Imperative.init

We can obtain the profile definitions by calling Imperative.init. It reads the command definitions and profile definitions for the Zowe commands and plugin commands. This is the original source of profile definitions for the team-config schema file and the old-style XXX_meta.yaml files.

  • Pros

    • The ProfileInfo API will be able to report missing arguments, even if no schema file exists.
    • The ProfileInfo API would have the means to create a "config init" type operation in the future.
  • Cons

    • Imperative.init does many other actions that are not required by the ProfileInfo API or its consuming apps.
      • Some of those actions may introduce undesired dependencies on the Zowe-CLI.
    • We would probably have to devise a means to scale-back, or partially replicate the functionality of Imperative.init.
      • A scaled-back version of Imperative.init could be time-consuming to implement.
      • We may introduce unexpected edge-case errors.
      • It is not yet clear that we can successfully achieve a scaled-back Imperative.init. Some research would be required.

Our current plans

We have partially implemented an approach for reading profile definitions from schema and meta files.

  • We will accept that customers must correctly perform some procedural steps to get started.
  • We will return some error indicator when we cannot obtain the definitions necessary to provide a list of missing arguments.

If you have a strong business case for either of the two suggested approaches, voice your opinion.
That could justify a change in our current direction, or give us an objective to replace our current approach in a future PI.

@jellypuno
Copy link
Contributor

@gejohnston What is your recommended approach? The PROs indicated in option 2 is attractive to me because it means that we are going to remove some processes that we implemented in ZE and rely on the SDK. But I am concerned about some of the CONs 😄

  • Imperative.init does many other actions that are not required by the ProfileInfo API or its consuming apps.
    • Some of those actions may introduce undesired dependencies on the Zowe-CLI.

Dependency meaning that the user needs to install Zowe CLI and the appropriate plugins before they can use ZE?

It is not yet clear that we can successfully achieve a scaled-back Imperative.init. Some research would be required.

If for example option 2 is chosen, Can we still start the development for our beta version? or does this mean that we need to pause and wait?

@gejohnston
Copy link
Member Author

@jellypuno Creating a dependency on the CLI is what we are worried about with an approach of using a scaled-back Imperative.init. It is not an expectation. I heard information in a meeting this morning that top levels of Product Management do NOT want ZE to depend on the CLI being installed. If our efforts to create a scaled-back version of Imperative.init for this API resulted in a dependency on the CLI, we would abandon the Imperative.init approach and resort to the disk-only approach.

Your interest in a scaled-back Imperative.init approach might justify some initial research for us. We may proceed with the disk-only approach for now, and if desireable, replace it with an Imperative-Init approach later when we have worked out any undesirable side-effects. I think the ProfileInfo API may hide most of such an implementation change, but some of the end-user limitations might change if we can later plug in a fully functional Imperative.init replacement. I think this would enable you to start development on your beta version sooner rather than later. Neither approach is implemented yet, but the disk-only approach will probably be faster to produce than the Imperative.init approach.

Some of our choices will also be driven by time-to-market concerns and Product Management priorities. We have not really discussed these particular alternatives with our own Product Management team yet.

@gejohnston
Copy link
Member Author

We welcome your feedback on how we present missing arguments to you

Our API has a couple of functions (mergeArgsForXXX) that return an array of known arguments (knownArgs) and an array of missing arguments (missingArgs). We want to give you the opportunity to influence how we interpret "missing" arguments.

Current implementation

If no value is specified for a required argument within the profile, but that argument has a default value, the argument is placed into the missingArgs array with its default value. The location of that argument will be marked as DEFAULT.

Before we return from mergeArgsForXXX, if the length of missingArgs is greater than Zero, we throw an exception containing an error code of MISSING_REQ_PROP. All data that we gathered will be available in knownArgs and missingArgs.

Since this approach is already implemented, this will remain the implementation, unless you voice a preference for different behavior.

Alternate implementation choice

Instead of placing such a default value in missingArgs, we could instead place the default value into knownArgs. Such an argument would still be marked as DEFAULT. Your program could then detect if all required arguments have some value by testing whether the length of missingArgs is zero.

Do you prefer this approach?

Do you want an exception to be thrown?

Regardless of whether we use the current or alternate implementation, do you want:

  • An exception to be thrown when missing arguments are identified, or
  • Do you prefer that the function ends successfully and your own code determines whether there are missing arguments based on the data that we return?

Ensuring that @jellypuno @VitGottwald and @t1m0thyj are notified.

@MikeBauerCA
Copy link

@gejohnston can this issue be closed?

@gejohnston
Copy link
Member Author

@MikeBauerCA I believe this issue can be closed. The material was reviewed by external teams. We made some changes to address feedback. An API with those changes was published and is being used by other teams. I should have closed this issue during the last merge of API changes, but I probably just lost track of this issue along the way.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
enhancement New feature or request team-profile
Projects
None yet
Development

No branches or pull requests

9 participants