V1.4 Bug Bash

[suiguoxin]

Bug Bash for release v1.4 #4988

Endgame

• INT deploy: 11/18
• Target Test Duration: 11/17 ~ 11/30

upgrade plan

• ne/rr: 12/2
• wu2: 12/9

---

Planned Items

multi-cluster

• P0 - Hybrid or multi-cluster management (SHAIIC) Multi Cluster Load Balance  #4929 @hzy46
  test owner: @yiyione @debuggy ( test in int/32 bed)
  test cases: Support job transfer #5082 (comment)
  issues: Browser warning: cookie will be soone treated as cross-site cookie against #5118

• #### Hived CPU Support CPU jobs on GPU nodes in HiveD scheduler #5075
  test owner: @hzy46
  test cases: use this config then submit cpu and gpu jobs on same node

Log Collection

• P1 Log collection PAI logging pipeline enhance #4992
  • API @Binyang2014
  • Front-end @AmberMsy
    • update API logic for Stdout / Stderr / Sdtout+Stderr button
    • update Browse log folder

test owner: @abuccts
test cases:

• Run command which will write disk heavily

• Test log url will be expired after 2 mins

• Click Stdout/Stderr/Stdout+Stderr button to get the corresponding log.

• Click Stdout/Stderr/Stdout+Stderr button and wait for 2 mins, the log content can still be displayed.

• When the current log file is less than 15K and there is a log.1 file, it will merge log and log.1 files and display the last 16K content. (log here refers to Stdout, Stderr, etc.)

• Click the Show All Logs button. A Dialog box will pop up, linking to all logs.

• Input invalid log name, should return 400

• Show all logs should download log file, Check open huge log file will not block frontend
  known issues:

• show all logs crash Open large log will crash browser tab #5119

• show all logs crash (webportal) Show all logs crash #5122

• all task retry logs
  ETA: 11.27

Marketplace

Data @debuggy

• P0 Move NFS to Azure Blob as backend -- ETA: Nov. 5
• P0 Upload Job output to Azure Blob ETA: 11.9
• P0 Download data from azure blob to local -- ETA: Nov. 9
• P1 Use Azure storage SDK for privacy -- ETA: 11.13
• P1 Refactor data use logic after change storage to blob ETA: 11.13
  test owner: @TobeyQin @mingliang's intern
  test cases:
  • Webpage layout and items display correctly.
  • All the link and button works.
  • Test data download
  • Test Create template logic and if it can successfully add template.
  • Test Azure storage URL is correct and accessible.
  • Test if all the jobs can finish successfully.
  • Test if can get output files when job finished.
    known issues:
  1. • Failed to fetch job log (Failed to fetch job logs #5094 )
  2. • Error no available connection strings Marketplace #104 PR in review
  3. • Cannot read property 'commands' of undefined [Marketplace Kubernetes issue f ix #105] (Cannot read property 'commands' of undefined openpaimarketplace#105) PR in review
  4. • covid chestxray dataset job failedMarketplace #106
• PAI relies on ntp #112, 113,117,118
  ETA: 11.27

Service Deployment -- @yiyione

• P0 Start Local Rest Server -- ETA: Nov. 2
• P0 Deployed Rest Server in PAI -- ETA: Nov. 5
• P0 Start database and save items into it ETA: 11.9
• P0 Register in PAI pylon PR: [Marketplace] Deploy scripts and config #5066 -- ETA: Nov. 10 PR review
• P0 Create marketplace item -- ETA: Nov. 12
  test owner: @debuggy @suiguoxin
  test cases: [Marketplace] Deploy scripts and config #5066 (comment)

Web Portal Experience

• P0 Job detail page cannot be searched by browser Job detail page cannot be searched by browser #4917 @debuggy ETA: 11/16 needs confirm
• P0 After click stop button successfully, the button is still enabled until the job becomes Stopping Job stop button no feedback after click successfully #5023 @debuggy ETA: 11/16 Disable stop button immediately after click #5079 in review
  test owner: @yqwang-ms
  test cases:

Admin Experience

• P1 Alert rules update: [Monitor] Alert System Refine #4810 @suiguoxin alert handler stop-job notice not clear to end user #5021
  test owner: @Binyang2014
  test cases:

• alert inhibition : PaiServicePodNotRunning / PaiServicePodNotRunning should not be fired when NodeNotReady is already fired

• alert severity render

• alert email templates: customized templates, kill-user-job-alert template render;

• P2 Deployment issues Deploy Issues #5001 @suiguoxin
  test owner: @hzy46
  test cases:

• deploy with master branch code & self-built dev-box image with dockerfile from master branch

Bug Fix

• Admin auth check issue in APIs: getGroupUserList, createGroup, creatUser, updateUserGroupList @suiguoxin
  test owner: @abuccts
  test cases:
• getGoupUserList (get /api/v2/group/:groupname/userlist):
  • test in AAD or basic AUTH mode
  • the response of the API with admin / non-admin token (200 / 401)
• createGroup (post /api/v2/group) :
  • test in AAD or basic AUTH mode
  • the response of the API with admin / non-admin token (201 / 403)
  • use getGroup (Get /api/v2/group/:groupname) API to check if the group is created (should not be created with non-admin token)
• createUser (post /api/v2/users):
  • test in basic AUTH mode
  • the response of the API with admin / non-admin token
  • use getUser (Get /api/v2/users/:username) API to check if the user is created
• updateUserGroupList (put /api/v2/users/:username/grouplist)
  • test in basic AUTH mode
  • the response of the API with admin / non-admin token
  • use getUser (Get /api/v2/users/:username) API to check if the user group list is updated (should not be updated with non-admin token)

issues:

• updateGroupUserList api not work #5120

• hived config issue caused by k8s coreDNS deployment CoreDNS pod will affect resource calculation for hived scheduler #5056 @Binyang2014 @hzy46 PR: Set coreDNS requests to zero #5071
  test owner: @suiguoxin
  test cases: