Skip to content

DLPX-89763 DLPX-86523 delphix-platform changes #477

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 3 commits into
base: develop
Choose a base branch
from

Conversation

justsanjeev
Copy link
Contributor

@justsanjeev justsanjeev commented Apr 18, 2024

Problem

CIS is looking or a single home directory filesystem mounted at the /home location, currently we have the home dataset is mounted on /export/home

Due to that we see the below issues in the CIS Report

  • (1.45) 7402 Status of the '/home' partition in the '/etc/fstab' file
  • (1.46) 13248 Status of Mount Partition '/home' using mount command
  • (1.47) 7403 Status of the 'nodev' mount option setting for the '/home partition' defined in the '/etc/ fstab' file
  • (1.48) 14601 Status of the 'nodev' option for '/home' partition using 'mount' command

Solution

  • Mounting the home dataset to /home.
  • Post upgrade creating a softlink /export/home to /home so that the exiting tests and dependencies have minimal impact.

We have now implemented the delphix-legacy-link.service for the /export/home -> /home creation, we earlier tried to do the sae in the delphix-platform.services and tested to implement the change with ExecStart and ExecStartPost. It dod not work and then this change was implemented to achieve the same after the delphix-platform.services are up completely.

Testing Done

-------Start of Testing of Build 11635 13 July 2025 ----------

Git ab-pre-push : 11635 - 🟠

Pre-checkin: Initial pre-checking job with above was failed and later passed with https://github.com/delphix/qa-infra/pull/2315

Successful Pre-chekin: https://selfservice-jenkins.eng-tools-prd.aws.delphixcloud.com/job/blackbox-self-service/183042/consoleFull - 🟢

Additional Automated Upgrade 2025.20.0 --> develop : https://selfservice-jenkins.eng-tools-prd.aws.delphixcloud.com/job/blackbox-chained/7870/consoleFull - 🟢

Manual Upgrade upgrade from dlpx-2025.3.0.0 --> develop : 🟢 | Engine : dlpx2025300.dlpxdc.co | Alive : 🟢
In manual upgrade we created an appdata dsource and created a vdb on the target engine, after upgrade our dsource was Active & the vdb was founf in Running state.
Screenshot 2025-07-14 at 5 31 43 PM

Scan Results
Created a Engine from the AMI published from the appliance job above

dc clone  -s ami-03c02c00e258efd64 dlpx-sanjeev.rohila-develop dlpx-dev-sr-03c02c00e258efd64

As we are now upgraded to Ubuntu 24.04 , there are changes in the CIS test, we have Ran the CIS Scans with the now with a new policy. we have some changes 7402 , 7403, 14601 and 13248 are not present in the new policy instead below 3 new controls closer to

  • 29044 : 🟢
  • 29045: 🟢
  • 29046: 🟢

CIS Scan Report

lsof output, no dependency on /export/home -

delphix@ip-10-110-199-125:~$ sudo lsof +f -- /export/home
COMMAND     PID    USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
delphix-s  1814    root  cwd    DIR   0,36       13    3 /home/delphix
delphix-s  1819    root  cwd    DIR   0,36       13    3 /home/delphix
delphix-s  1820    root  cwd    DIR   0,36       13    3 /home/delphix
delphix-s  1827    root  cwd    DIR   0,36       13    3 /home/delphix
delphix-s  1828    root  cwd    DIR   0,36       13    3 /home/delphix
delphix-s  1829    root  cwd    DIR   0,36       13    3 /home/delphix
delphix-s  1831    root  cwd    DIR   0,36       13    3 /home/delphix
delphix-s  1832    root  cwd    DIR   0,36       13    3 /home/delphix
delphix-s  1834    root  cwd    DIR   0,36       13    3 /home/delphix
delphix-s  1836    root  cwd    DIR   0,36       13    3 /home/delphix
delphix-s  1838    root  cwd    DIR   0,36       13    3 /home/delphix
delphix-s  1840    root  cwd    DIR   0,36       13    3 /home/delphix
delphix-s  1841    root  cwd    DIR   0,36       13    3 /home/delphix
delphix-s  1842    root  cwd    DIR   0,36       13    3 /home/delphix
delphix-s  1844    root  cwd    DIR   0,36       13    3 /home/delphix
delphix-s  1845    root  cwd    DIR   0,36       13    3 /home/delphix
delphix-s  1846    root  cwd    DIR   0,36       13    3 /home/delphix
delphix-s  1847    root  cwd    DIR   0,36       13    3 /home/delphix
delphix-s  1849    root  cwd    DIR   0,36       13    3 /home/delphix
delphix-s  1851    root  cwd    DIR   0,36       13    3 /home/delphix
delphix-s  1853    root  cwd    DIR   0,36       13    3 /home/delphix
delphix-s  1854    root  cwd    DIR   0,36       13    3 /home/delphix
delphix-s  1859    root  cwd    DIR   0,36       13    3 /home/delphix
nfs_delet  1860    root  cwd    DIR   0,36       13    3 /home/delphix
timeout    1892    root  cwd    DIR   0,36       13    3 /home/delphix
timeout    1895    root  cwd    DIR   0,36       13    3 /home/delphix
timeout    1897    root  cwd    DIR   0,36       13    3 /home/delphix
timeout    1898    root  cwd    DIR   0,36       13    3 /home/delphix
vmstat     1909    root  cwd    DIR   0,36       13    3 /home/delphix
timeout    1920    root  cwd    DIR   0,36       13    3 /home/delphix
arcstat    1924    root  cwd    DIR   0,36       13    3 /home/delphix
mpstat     1967    root  cwd    DIR   0,36       13    3 /home/delphix
iostat     1970    root  cwd    DIR   0,36       13    3 /home/delphix
arcstat    1978    root  cwd    DIR   0,36       13    3 /home/delphix
timeout    1999    root  cwd    DIR   0,36       13    3 /home/delphix
timeout    2002    root  cwd    DIR   0,36       13    3 /home/delphix
zpool      2020    root  cwd    DIR   0,36       13    3 /home/delphix
zpool      2031    root  cwd    DIR   0,36       13    3 /home/delphix
sleep      2041    root  cwd    DIR   0,36       13    3 /home/delphix
sleep      2143    root  cwd    DIR   0,36       13    3 /home/delphix
bpftrace   2153    root  cwd    DIR   0,36       13    3 /home/delphix
nfs_delet  2154    root  cwd    DIR   0,36       13    3 /home/delphix
sleep      3339    root  cwd    DIR   0,36       13    3 /home/delphix
java       7179    root  cwd    DIR   0,36       13    3 /home/delphix
python3    9361    root  cwd    DIR   0,36       13    3 /home/delphix
python3    9364    root  cwd    DIR   0,36       13    3 /home/delphix
python3    9368    root  cwd    DIR   0,36       13    3 /home/delphix
python3    9372    root  cwd    DIR   0,36       13    3 /home/delphix
sleep     14453    root  cwd    DIR   0,36       13    3 /home/delphix
sleep     14523    root  cwd    DIR   0,36       13    3 /home/delphix
sleep     14524    root  cwd    DIR   0,36       13    3 /home/delphix
sleep     14652    root  cwd    DIR   0,36       13    3 /home/delphix
sleep     15600    root  cwd    DIR   0,36       13    3 /home/delphix
sleep     15604    root  cwd    DIR   0,36       13    3 /home/delphix
sleep     15608    root  cwd    DIR   0,36       13    3 /home/delphix
sleep     15612    root  cwd    DIR   0,36       13    3 /home/delphix
sleep     15616    root  cwd    DIR   0,36       13    3 /home/delphix
sleep     15620    root  cwd    DIR   0,36       13    3 /home/delphix
sleep     15624    root  cwd    DIR   0,36       13    3 /home/delphix
script    15723 delphix  cwd    DIR   0,36       13    3 /home/delphix
bash      15734 delphix  cwd    DIR   0,36       13    3 /home/delphix
sleep     15764    root  cwd    DIR   0,36       13    3 /home/delphix
sleep     15768    root  cwd    DIR   0,36       13    3 /home/delphix
sudo      15769    root  cwd    DIR   0,36       13    3 /home/delphix
sudo      15770    root  cwd    DIR   0,36       13    3 /home/delphix
lsof      15771    root  cwd    DIR   0,36       13    3 /home/delphix
lsof      15772    root  cwd    DIR   0,36       13    3 /home/delphix
delphix@ip-10-110-199-125:~$ 

Softlink /export/home -> /home - 🟢

delphix@ip-10-110-199-125:~$  ls -ltr /export
total 1
lrwxrwxrwx 1 root root 5 Jul 14 11:34 home -> /home
delphix@ip-10-110-199-125:~$ 

-------End of Testing of Build 11635 13 July 2025----------

@justsanjeev justsanjeev force-pushed the dlpx/pr/justsanjeev/ef30769e-a58a-4475-af32-174db6d07951 branch from ec9dada to 317e58e Compare April 18, 2024 14:34
@justsanjeev justsanjeev force-pushed the dlpx/pr/justsanjeev/ef30769e-a58a-4475-af32-174db6d07951 branch from 317e58e to 68118d0 Compare June 24, 2024 09:31
@justsanjeev justsanjeev force-pushed the dlpx/pr/justsanjeev/ef30769e-a58a-4475-af32-174db6d07951 branch from 68118d0 to e56b7de Compare July 2, 2024 11:54
@justsanjeev justsanjeev force-pushed the dlpx/pr/justsanjeev/ef30769e-a58a-4475-af32-174db6d07951 branch 4 times, most recently from a4f876c to d356ef1 Compare July 11, 2024 10:29
@justsanjeev justsanjeev force-pushed the dlpx/pr/justsanjeev/ef30769e-a58a-4475-af32-174db6d07951 branch 5 times, most recently from 38fba97 to 28cab6b Compare July 16, 2024 07:08
@justsanjeev justsanjeev force-pushed the dlpx/pr/justsanjeev/ef30769e-a58a-4475-af32-174db6d07951 branch 2 times, most recently from b1d61e7 to 0c718f3 Compare September 25, 2024 17:25
@justsanjeev justsanjeev force-pushed the dlpx/pr/justsanjeev/ef30769e-a58a-4475-af32-174db6d07951 branch 2 times, most recently from b984a71 to f389cbe Compare October 7, 2024 10:17
@justsanjeev justsanjeev force-pushed the dlpx/pr/justsanjeev/ef30769e-a58a-4475-af32-174db6d07951 branch 2 times, most recently from 64091f7 to 5022542 Compare October 22, 2024 11:48
@justsanjeev justsanjeev force-pushed the dlpx/pr/justsanjeev/ef30769e-a58a-4475-af32-174db6d07951 branch from 5022542 to 744dc46 Compare November 4, 2024 07:54
@justsanjeev justsanjeev marked this pull request as ready for review November 5, 2024 10:42
@justsanjeev justsanjeev force-pushed the dlpx/pr/justsanjeev/ef30769e-a58a-4475-af32-174db6d07951 branch 3 times, most recently from d39c466 to 52eee14 Compare November 25, 2024 07:26
@justsanjeev justsanjeev force-pushed the dlpx/pr/justsanjeev/ef30769e-a58a-4475-af32-174db6d07951 branch 3 times, most recently from b5743b5 to 3a9629e Compare December 3, 2024 08:11
@justsanjeev justsanjeev force-pushed the dlpx/pr/justsanjeev/ef30769e-a58a-4475-af32-174db6d07951 branch from 4e5c764 to d0ba9f9 Compare December 3, 2024 11:33
@justsanjeev justsanjeev force-pushed the dlpx/pr/justsanjeev/ef30769e-a58a-4475-af32-174db6d07951 branch from d0ba9f9 to 43d194d Compare January 16, 2025 06:30
@justsanjeev justsanjeev force-pushed the dlpx/pr/justsanjeev/ef30769e-a58a-4475-af32-174db6d07951 branch 5 times, most recently from 24f6877 to 5a17a79 Compare April 15, 2025 09:26
@justsanjeev
Copy link
Contributor Author

justsanjeev commented Apr 16, 2025

We have made the latest changes here and added the below logic to delphix-platform.service to take care of the /export/home mount point and soft link /export/home -> /home

# Create /export/home symlink to /home if it doesn't already exist
ExecStartPost=/bin/sh -c '\
  echo "Ensuring /export/home is a symlink to /home..."; \
  if mountpoint -q /export/home; then \
    echo "/export/home is a mountpoint, unmounting..."; \
    umount /export/home || { echo "Failed to unmount /export/home"; exit 1; }; \
  fi; \
  if [ -e /export/home ] && [ ! -L /export/home ]; then \
    echo "Removing existing /export/home directory..."; \
    rm -rf /export/home; \
  fi; \
  if [ ! -d /export ]; then \
    mkdir /export; \
  fi; \
  if [ -d /export ] && [ ! -L /export/home ]; then \
    echo "Creating symlink: /export/home -> /home"; \
    ln -s /home /export/home; \
  fi;'

We this change we got a build : appliance-build-orchestrator-pre-push/10842.

  • Here we have the successful upgrade tests.
  • But to verify @sebroy 's comment, I again the same testing

-- Created an app data unstructured vdb on a target
-- Upgraded the engine
-- delphix-mgmt did not come up due to delphix-platform
Screenshot 2025-04-16 at 9 16 19 PM

Also my /var/log/syslog show

Apr 15 12:04:11 ip-10-110-212-88 dynamic-debug[156701]: init/main.c:1218 [main]initcall_blacklisted =p "initcall %s blacklisted\012"
Apr 15 12:04:11 ip-10-110-212-88 dynamic-debug[156701]: init/main.c:1179 [main]initcall_blacklist =p "blacklisting initcall %s\012"
Apr 15 12:04:11 ip-10-110-212-88 sh[156702]: Ensuring /export/home is a symlink to /home...
Apr 15 12:04:11 ip-10-110-212-88 sh[156702]: /export/home is a mountpoint, unmounting...
Apr 15 12:04:11 ip-10-110-212-88 sh[156704]: umount: /export/home: target is busy.
Apr 15 12:04:11 ip-10-110-212-88 sh[156702]: Failed to unmount /export/home
Apr 15 12:04:11 ip-10-110-212-88 systemd[1]: delphix-platform.service: Control process exited, code=exited, status=1/FAILURE
Apr 15 12:04:11 ip-10-110-212-88 systemd[1]: delphix-platform.service: Failed with result 'exit-code'.
Apr 15 12:04:11 ip-10-110-212-88 systemd[1]: Failed to start Delphix Appliance Platform Service.

Upon executing below

delphix@ip-10-110-212-88:~$ sudo systemctl enable delphix-platform.service
delphix@ip-10-110-212-88:~$ sudo systemctl restart delphix-mgmt

And my log shows that the services has started successfully

Apr 15 12:37:42 ip-10-110-212-88 dynamic-debug[165503]: init/main.c:1179 [main]initcall_blacklist =p "blacklisting initcall %s\012"
Apr 15 12:37:42 ip-10-110-212-88 sh[165504]: Ensuring /export/home is a symlink to /home...
Apr 15 12:37:42 ip-10-110-212-88 sh[165504]: /export/home is a mountpoint, unmounting...
Apr 15 12:37:42 ip-10-110-212-88 systemd[1]: export-home.mount: Succeeded.
Apr 15 12:37:42 ip-10-110-212-88 systemd[164396]: export-home.mount: Succeeded.
Apr 15 12:37:42 ip-10-110-212-88 sh[165504]: Removing existing /export/home directory...
Apr 15 12:37:42 ip-10-110-212-88 sh[165504]: Creating symlink: /export/home -> /home
Apr 15 12:37:42 ip-10-110-212-88 systemd[1]: Finished Delphix Appliance Platform Service.
Apr 15 12:37:42 ip-10-110-212-88 systemd[1]: Starting Delphix management service...

Setup Present for the same setup: dlpx-qa-2025200-qar-170885-766f97e2.dlpxdc.co
Upgraded from: 2025.2.0.0
Upgrade Tar: s3://dev-de-images/builds/jenkins-selfservice/appliance-build/develop/pre-push/4235/upgrade-artifacts/internal-qa.upgrade.tar
An Engine retained in the Error State: dlpx-qa-2025200-qar-170888-766f97e2.dlpxdc.co

@justsanjeev
Copy link
Contributor Author

justsanjeev commented Apr 16, 2025

Now here, we have created an alternative PR where we have added the logic to cleanup the /export/home mount point and creating the soft link /export/home -> /home.

We have added another service delphix-legacy-link.service

This Build is Validated : https://selfservice-jenkins.eng-tools-prd.aws.delphixcloud.com/job/appliance-build-orchestrator-pre-push/10846/
ManualUpgrade with an Unstructured database mounted:
Manual Upgraded Engine, with deffered and Full Upgrade : dlpx-qa-2025200-qar-170959-766f97e2.dlpxdc.co
Failed Pre Checkin Test Rebuild: pre_checkin

@justsanjeev
Copy link
Contributor Author

Post changing my code as per above comments, I have Build appliance-build-orchestrator-pre-push/10897 running.

@justsanjeev justsanjeev force-pushed the dlpx/pr/justsanjeev/ef30769e-a58a-4475-af32-174db6d07951 branch from 27a3f5e to 8f0067f Compare April 22, 2025 06:00
Copy link
Contributor

@sebroy sebroy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we're almost there with this. The only remaining detail is the mechanism used to run one-time boot-time logic.

@justsanjeev justsanjeev force-pushed the dlpx/pr/justsanjeev/ef30769e-a58a-4475-af32-174db6d07951 branch from f2356e1 to e003914 Compare May 19, 2025 15:54
@justsanjeev justsanjeev force-pushed the dlpx/pr/justsanjeev/ef30769e-a58a-4475-af32-174db6d07951 branch 7 times, most recently from 1c3baff to e22f588 Compare June 4, 2025 17:01
@justsanjeev justsanjeev force-pushed the dlpx/pr/justsanjeev/ef30769e-a58a-4475-af32-174db6d07951 branch from e22f588 to 26210f6 Compare June 30, 2025 16:20
@justsanjeev justsanjeev force-pushed the dlpx/pr/justsanjeev/ef30769e-a58a-4475-af32-174db6d07951 branch 2 times, most recently from 8b5f0fb to a5ef067 Compare July 13, 2025 07:15
debian/postinst Outdated
Comment on lines 127 to 129
if [[ -f "/var/lib/delphix-platform/export-home" ]]; then
chmod 755 "/var/lib/delphix-platform/export-home"
fi
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This shouldn't be necessary. Note that there is no equivalent code needed for the other executables delivered by this package. You should be able to set the permissions on the file directly in the repo.

echo 'Ensuring /export/home is a symlink to /home...'
if [ -e /export/home ]; then
echo 'Removing existing /export/home directory...'
rm -rf /export/home
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems risky. If there's some bug in our code and we fail to unmount home directories for whatever reason, this will delete all home directories. At this point, if everything works as designed, /export/home should be an empty directory (it's the mountpoint for a filesystem that should have been unmounted already), and so rmdir should succeed.

…rt/home

Adding the mount point logic back to delphix-platform.sh script
Changing as per Seb-s comment to alter the softlink logic.
Changing the code to previous version to validate the comments.
Changing the delphix-legacy-link.service to perform the same steps next reboot like deferred upgrade.
Creating a different service for taking care of soft /export/home -> /home
DLPX-89763 DLPX-86523 delphix-platform changes
Moved the code to create symlink to different script
Removing the autofs handling as per the concerns raised by Seb and our validation
Removing the softlink loginc from main.yml

PR URL: https://www.github.com/delphix/delphix-platform/pull/477
@justsanjeev justsanjeev force-pushed the dlpx/pr/justsanjeev/ef30769e-a58a-4475-af32-174db6d07951 branch from eba0198 to b36fe93 Compare July 21, 2025 11:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

3 participants