IBM User Build - Cannot download log files from logs sub-folders #296

FALLAI-Denis · 2022-11-29T18:38:22Z

Development environment used

Z Open Editor version: 3.0.0
Editor Platform
- Visual Studio Code
- Red Hat CodeReady Workspaces
- Eclipse Che
- Standalone Theia
Editor Platform Version: 1.72.2
Operating System: Windows 10
Java Version: 11
Related to RSE API?
- RSE API Plugin version:
- Zowe CLI version:
- Node.js version:
Logs attached (see here how to get them): yes/no

Problem Description

ZOE seems to have trouble identifying log files to download after a user build.

If we don't specify a filter with the logFilePatterns parameters, it downloads all the files of all the builds it finds, and ends up with an error:

profiles:
  - name: dbb-userbuild
    type: dbb
    settings:
      application: CEAB3.MVP2
      command: "$DBB_HOME/bin/groovyz"
      buildScriptPath: "/var/DBB/PROD/SCRIPTS/buildBPCE.groovy"
      buildScriptArgs:
        - "--userBuild"
        - "--workspace ${zopeneditor.userbuild.userSettings.dbbWorkspace}"
        - "--application ${application}"
        - "--hlq ${zopeneditor.userbuild.userSettings.dbbHlq}.${application}"
        - "--outDir ${zopeneditor.userbuild.userSettings.dbbLogDir}"                 <== log root folder
      additionalDependencies:
        - application-conf
        - .build
        - "src\\**\\${buildFile}.build"

Running User Build...
Identified the local root workspace path as c:\Trv\DepotsGIT\ceab3-environn-outinfr-central-mvp2
---------------------
Checking dbb profile settings defined in your repository's zapp.yaml or zapp.json
User setting dbbDefaultZappProfile: **dbb-userbuild**
Will read settings from ZAPP profile called dbb-userbuild defined in User Settings
    application: CEAB3.MVP2
    command: $DBB_HOME/bin/groovyz
    buildScriptArgs: --userBuild,--workspace /u/J070188/dbb,--application CEAB3.MVP2,--hlq J070188.DBB.CEAB3.MVP2,--outDir /u/J070188/dbb/logs
    buildScriptPath: /var/DBB/PROD/SCRIPTS/buildBPCE.groovy
    additionalDependencies: application-conf,.build,src\**\S9TL1A.cbl.build
    logFilePatterns: **Optional setting is missing**
---------------------
Checking user settings...
User settings should be defined in your VS Code or Theia Settings under zopeneditor.userbuild.userSettings
    dbbWorkspace: /u/J070188/dbb
    dbbHlq: J070188.DBB
    dbbLogDir: /u/J070188/dbb/logs
    dbbDefaultZappProfile: dbb-userbuild
    localLogDir: .reports
[...]
-------------------------------------------------------------------------------------------------------------------------
** Writing build report data to /u/J070188/dbb/logs/build.20221129.191008.010/BuildReport.json
** Writing build report to /u/J070188/dbb/logs/build.20221129.191008.010/BuildReport.html
-------------------------------------------------------------------------------------------------------------------------
** Build ended at Tue Nov 29 19:10:16 CET 2022
** Build State : CLEAN
** Total files in build list : 1
** Total build time  : 7.956 seconds
-------------------------------------------------------------------------------------------------------------------------
** Build finished
$ exit $?
Looking for user build logs in "/u/J070188/dbb/logs".       <== given value on --outDir parameter
Downloading logs to c:\Trv\DepotsGIT\ceab3-environn-outinfr-central-mvp2\.reports
Downloading log file /u/J070188/dbb/logs/.userbuilddependencies. Binary: false
Downloading log file /u/J070188/dbb/logs/build.20221129.162418.024. Binary: false
Downloading log file /u/J070188/dbb/logs/build.20221129.170139.001. Binary: false
Downloading log file /u/J070188/dbb/logs/build.20221129.171827.018. Binary: false
Downloading log file /u/J070188/dbb/logs/build.20221129.183526.035. Binary: false
Downloading log file /u/J070188/dbb/logs/build.20221129.184044.040. Binary: false
Downloading log file /u/J070188/dbb/logs/build.20221129.185218.052. Binary: false
Downloading log file /u/J070188/dbb/logs/build.20221129.191008.010. Binary: false   <== only this folder in concerned by current build
Error: Failed to download logs: Error: z/OSMF REST API Error:       <== unidentified error
Rest API failure with HTTP(S) status 500
category: 6
rc:       8
reason:   92078512
message:  file lock via fcntl() failed
details: 
  - EDC5121I Invalid argument. (errno2=0x055501B0)

/u/J070188/dbb/logs/build.<date>.<time>.<mmm> are folders... ZOE try to download files...

Specifying a filter with the File Patterns log parameters does not download any logs:

profiles:
  - name: dbb-userbuild
    type: dbb
    settings:
      application: CEAB3.MVP2
      command: "$DBB_HOME/bin/groovyz"
      buildScriptPath: "/var/DBB/PROD/SCRIPTS/buildBPCE.groovy"
      buildScriptArgs:
        - "--userBuild"
        - "--workspace ${zopeneditor.userbuild.userSettings.dbbWorkspace}"
        - "--application ${application}"
        - "--hlq ${zopeneditor.userbuild.userSettings.dbbHlq}.${application}"
        - "--outDir ${zopeneditor.userbuild.userSettings.dbbLogDir}"
      additionalDependencies:
        - application-conf
        - .build
        - "src\\**\\${buildFile}.build"
      logFilePatterns:
        - "${buildFile.basename}*.log"
        - "BuildReport.*"

Running User Build...
Identified the local root workspace path as c:\Trv\DepotsGIT\ceab3-environn-outinfr-central-mvp2
---------------------
Checking dbb profile settings defined in your repository's zapp.yaml or zapp.json
User setting dbbDefaultZappProfile: **dbb-userbuild**
Will read settings from ZAPP profile called dbb-userbuild defined in User Settings
    application: CEAB3.MVP2
    command: $DBB_HOME/bin/groovyz
    buildScriptArgs: --userBuild,--workspace /u/J070188/dbb,--application CEAB3.MVP2,--hlq J070188.DBB.CEAB3.MVP2,--outDir /u/J070188/dbb/logs
    buildScriptPath: /var/DBB/PROD/SCRIPTS/buildBPCE.groovy
    additionalDependencies: application-conf,.build,src\**\S9TL1A.cbl.build
    logFilePatterns: S9TL1A*.log,BuildReport.*
---------------------
Checking user settings...
User settings should be defined in your VS Code or Theia Settings under zopeneditor.userbuild.userSettings
    dbbWorkspace: /u/J070188/dbb
    dbbHlq: J070188.DBB
    dbbLogDir: /u/J070188/dbb/logs
    dbbDefaultZappProfile: dbb-userbuild
    localLogDir: .reports
[...]
** Writing build report data to /u/J070188/dbb/logs/build.20221129.192630.026/BuildReport.json
** Writing build report to /u/J070188/dbb/logs/build.20221129.192630.026/BuildReport.html
-------------------------------------------------------------------------------------------------------------------------
** Build ended at Tue Nov 29 19:26:36 CET 2022
** Build State : CLEAN
** Total files in build list : 1
** Total build time  : 6.417 seconds
-------------------------------------------------------------------------------------------------------------------------
** Build finished
$ exit $?
Looking for user build logs in "/u/J070188/dbb/logs".
Will download logs matching the following patterns: S9TL1A*.log,BuildReport.*
Downloading logs to c:\Trv\DepotsGIT\ceab3-environn-outinfr-central-mvp2\.reports
Successfully finished downloading of 0 log files to c:\Trv\DepotsGIT\ceab3-environn-outinfr-central-mvp2\.reports.   <== 0 log found

List from ssh session:

J070188:/u/J070188/dbb/logs: >find . -name S9TL1A* -o -name BuildReport*
./build.20221129.162418.024/BuildReport.json
./build.20221129.162418.024/BuildReport.html
./build.20221129.185218.052/logs/SIRIS/S9TL1A.log
./build.20221129.185218.052/logs/SIRIS/S9TL1A.lked.log
./build.20221129.185218.052/BuildReport.json
./build.20221129.185218.052/BuildReport.html
./build.20221129.191008.010/logs/SIRIS/S9TL1A.log
./build.20221129.191008.010/logs/SIRIS/S9TL1A.lked.log
./build.20221129.191008.010/BuildReport.json
./build.20221129.191008.010/BuildReport.html
./build.20221129.192630.026/logs/SIRIS/S9TL1A.log            <== last build run
./build.20221129.192630.026/logs/SIRIS/S9TL1A.lked.log    <== last build run
./build.20221129.192630.026/BuildReport.json                     <== last build run
./build.20221129.192630.026/BuildReport.html                    <== last build run

profiles:
  - name: dbb-userbuild
    type: dbb
    settings:
      application: CEAB3.MVP2
      command: "$DBB_HOME/bin/groovyz"
      buildScriptPath: "/var/DBB/PROD/SCRIPTS/buildBPCE.groovy"
      buildScriptArgs:
        - "--userBuild"
        - "--workspace ${zopeneditor.userbuild.userSettings.dbbWorkspace}"
        - "--application ${application}"
        - "--hlq ${zopeneditor.userbuild.userSettings.dbbHlq}.${application}"
        - "--outDir ${zopeneditor.userbuild.userSettings.dbbLogDir}"
      additionalDependencies:
        - application-conf
        - .build
        - "src\\**\\${buildFile}.build"
      logFilePatterns:
        - "${zopeneditor.userbuild.userSettings.dbbWorkspace}/**/${buildFile.basename}*.log"
        - "${zopeneditor.userbuild.userSettings.dbbWorkspace}/**/BuildReport.*"

** Writing build report data to /u/J070188/dbb/logs/build.20221129.201336.013/BuildReport.json
** Writing build report to /u/J070188/dbb/logs/build.20221129.201336.013/BuildReport.html
-------------------------------------------------------------------------------------------------------------------------
** Build ended at Tue Nov 29 20:13:42 CET 2022
** Build State : CLEAN
** Total files in build list : 1
** Total build time  : 6.545 seconds
-------------------------------------------------------------------------------------------------------------------------
** Build finished
$ exit $?
Looking for user build logs in "/u/J070188/dbb/logs".
Will download logs matching the following patterns: /u/J070188/dbb/**/S9TL1A*.log,/u/J070188/dbb/**/BuildReport.*
Downloading logs to c:\Trv\DepotsGIT\ceab3-environn-outinfr-central-mvp2\.reports
Successfully finished downloading of 0 log files to c:\Trv\DepotsGIT\ceab3-environn-outinfr-central-mvp2\.reports.   <== 0 log found

The text was updated successfully, but these errors were encountered:

FALLAI-Denis · 2022-11-29T19:41:32Z

The problem has been identified...

The people in charge of implementing DBB on our site have modified the access paths for storing logs to meet constraints related to our context:

Original version (zAppbuil 1.0.3)

build.groovy

if (props.userBuild) {	// do not create a subfolder for user builds
  props.buildOutDir = "${props.outDir}" as String }
else {// validate createBuildOutputSubfolder build property
  props.buildOutDir = ((props.createBuildOutputSubfolder && props.createBuildOutputSubfolder.toBoolean()) ? "${props.outDir}/${props.applicationBuildLabel}" : "${props.outDir}") as String
}

Cobol.groovy

File logFile = new File(props.userBuild ? "${props.buildOutDir}/${member}.log": "${props.buildOutDir}/${member}.cobol.log")`

Modified version

build.groovy

if (props.userBuild) {	// do not create a subfolder for user builds
  //props.buildOutDir = "${props.outDir}" as String
  props.buildOutDir = ((props.createBuildOutputSubfolder && props.createBuildOutputSubfolder.toBoolean()) ? "${props.outDir}/${props.applicationBuildLabel}" : "${props.outDir}") as String
}
else {// validate createBuildOutputSubfolder build property
  props.buildOutDir = ((props.createBuildOutputSubfolder && props.createBuildOutputSubfolder.toBoolean()) ? "${props.outDir}/${props.applicationBuildLabel}" : "${props.outDir}") as String
}

Cobol.groovy

File logFile = new File( props.userBuild ? "${props.buildOutDir}/logs/$systemName/${member}.log": "${props.buildOutDir}/logs/$systemName/${member}.cobol .log")

This demonstrates that IBM User Build of ZOE is in strong adherence with zAppbuild and that any modification made to zAppbuild breaks IBM User Build...

There should be more independence between IBM User Build and zAppbuild.

FALLAI-Denis · 2022-11-29T19:57:38Z

After restoring the build.groovy script to its initial state for managing userBuild logs:

** Writing build report data to /u/J070188/dbb/logs/BuildReport.json
** Writing build report to /u/J070188/dbb/logs/BuildReport.html
-------------------------------------------------------------------------------------------------------------------------
** Build ended at Tue Nov 29 20:51:33 CET 2022
** Build State : CLEAN
** Total files in build list : 1
** Total build time  : 6.520 seconds
-------------------------------------------------------------------------------------------------------------------------
** Build finished
$ exit $?
Looking for user build logs in "/u/J070188/dbb/logs".
Will download logs matching the following patterns: S9TL1A*.log,BuildReport.*
Downloading logs to c:\Trv\DepotsGIT\ceab3-environn-outinfr-central-mvp2\.reports
Downloading log file /u/J070188/dbb/logs/BuildReport.json. Binary: true
Downloading log file /u/J070188/dbb/logs/BuildReport.html. Binary: true
Successfully finished downloading of 2 log files to c:\Trv\DepotsGIT\ceab3-environn-outinfr-central-mvp2\.reports.

There remains a problem to retrieve the logs of the built program because we cannot store them at the root of the "log" folder (constraint of our context):

phaumer · 2022-12-02T01:55:36Z

@FALLAI-Denis currently we can only apply the glob files to the files located directly in the logs folder and not its sub-folders, yet. We need to fix our documentation to be clear about this and file this one as an enhancement to evaluate any sub-directories as well in the future. As it is a remote system there will be a performance cost, though.

phaumer · 2023-03-10T00:33:42Z

GLOB patterns will now evaluate sub-directories on USS as well in v3.1.0.

phaumer added the enhancement New feature or request label Dec 2, 2022

phaumer changed the title ~~IBM User Build - Various problems downloading logs after running the build~~ IBM User Build - Cannot download log files from logs sub-folders Dec 2, 2022

phaumer closed this as completed Mar 10, 2023

phaumer mentioned this issue Jul 26, 2023

Error while downloading logfiles after DBB userbuild #353

Open

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

IBM User Build - Cannot download log files from logs sub-folders #296

IBM User Build - Cannot download log files from logs sub-folders #296

FALLAI-Denis commented Nov 29, 2022 •

edited

FALLAI-Denis commented Nov 29, 2022

FALLAI-Denis commented Nov 29, 2022

phaumer commented Dec 2, 2022 •

edited

phaumer commented Mar 10, 2023

IBM User Build - Cannot download log files from logs sub-folders #296

IBM User Build - Cannot download log files from logs sub-folders #296

Comments

FALLAI-Denis commented Nov 29, 2022 • edited

Development environment used

Problem Description

FALLAI-Denis commented Nov 29, 2022

FALLAI-Denis commented Nov 29, 2022

phaumer commented Dec 2, 2022 • edited

phaumer commented Mar 10, 2023

FALLAI-Denis commented Nov 29, 2022 •

edited

phaumer commented Dec 2, 2022 •

edited