PWX-32428,PWX-27672: shared mounts fix for PKS and privileged:false

[zoxpx]

• fixes shared mounts for PKS (PWX-32428, regression introduced via PWX-31842)
• fixed shared mounts for privileged:false (PWX-27672)

What this PR does / why we need it:

• PWX-32428: Fixes regression introduced w/ PWX-31842
  • we are rolling back most of the PWX-31842: PKS mounts-leak fix #1081 changes, since they break shared/sharedV4 for PKS
• PWX-27672: Fixed shared/sharedV4 volumes when privileged:false annotation used
• new limitation: portworx.io/privileged: true/false annotation cannot be used unless on px-3.0.1 (or higher)
• new pksVolumeInfo.mountPropagation tweak can override volume's propagation (value - removes propagation setting)

FIX:

• we are taking advantage of the new mounts-rules changes introduced w/ px-3.0.1
• removing the :shared flag for -v /var/lib/osd:/var/lib/osd:shared
• adding new -v /var/lib/osd/pxns:/var/lib/osd/pxns:shared and -v /var/lib/osd/mounts:/var/lib/osd/mounts:shared mounts

Which issue(s) this PR fixes (optional)
Closes # PWX-32428
Also follow-up fix for PWX-27672

Special notes for your reviewer:

note: using https://github.com/libopenstorage/operator/tree/PWX-27672_privileged_switch_for_oci-mons as a base-branch

[zoxpx]

This is odd, unrelated TestMonitoringMetricsEnabled started failing on Travis-checks (this PR did not modify that part of the code)

• please ignore for now, I'll recheck everything once base-branch is committed, and this PR rebased off of master-branch

[pureneelesh]

sharedv4 mount changes look good to me

[zoxpx]

NOTE -- here's the result of the actual test on a PKS system:

• deployment on px-3.0.0
  • note, we're using same mountpoints as before

kind: Pod
spec:
  containers:
  - name: portworx
    image: docker.io/portworx/oci-monitor:3.0.0
    volumeMounts:
    - mountPath: /var/lib/osd/log
      name: pxlogs
    - mountPath: /var/lib/osd
      mountPropagation: Bidirectional
      name: varlibosd
  volumes:
  - hostPath:
      path: /var/vcap/store/lib/osd/log
      type: ""
    name: pxlogs
  - hostPath:
      path: /var/lib/osd
      type: ""
    name: varlibosd

• deployment on px-4.0.0 (a.k.a. master)
  • note, using non-shared /var/lib/osd and 2 extra shared mountpoint via params

kind: Pod
spec:
  containers:
  - name: portworx
    image: docker.io/zoxpx/oci-monitor:4.0.0-dev
    args:
    - ...
    - -v
    - /var/lib/osd/pxns:/var/lib/osd/pxns:shared
    - -v
    - /var/lib/osd/mounts:/var/lib/osd/mounts:shared
    volumeMounts:
    - mountPath: /var/lib/osd
      name: varlibosd
  volumes:
  - hostPath:
      path: /var/vcap/store/lib/osd
      type: ""
    name: varlibosd

[zoxpx]

note, code rebased off of latest master

[codecov]

Codecov Report

Patch coverage: 95.83% and project coverage change: +0.05% 🎉

Comparison is base (852a14d) 75.79% compared to head (1527ed7) 75.84%.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1181      +/-   ##
==========================================
+ Coverage   75.79%   75.84%   +0.05%     
==========================================
  Files          64       64              
  Lines       18123    18152      +29     
==========================================
+ Hits        13736    13768      +32     
+ Misses       3411     3408       -3     
  Partials      976      976              

Files Changed	Coverage Δ
pkg/migration/generate.go	88.82% <ø> (ø)
drivers/storage/portworx/deployment.go	95.90% <95.83%> (+0.33%)	⬆️

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[zoxpx]

Thanks for the review Neelesh and Piyush -- merging the PR, will cherry-pick into px-rel-23.7.0 via #1185