Introduce debug subcommand by nishakm · Pull Request #842 · tern-tools/tern

nishakm · 2020-12-07T23:38:50Z

Fixes #660

The "debug" option includes the ability to test out
script in base.yml and to inspect the filesystem at a given
layer index.

To inspect a layer's manifested filesystem run:
tern debug -i <image:tag> --step
This will set up an interactive session where the user will be
asked for a layer to inspect. Once chosen, Tern will mount the
layers and provide instructions to inspect the filesystem or or
chroot into it.
To recover the filesystem or clean up run:
tern debug --recover

To run a script from base.yml run:
tern debug -i <image:tag> --keys base <package_manager> <names/version/etc>
keys are the keys in base.yml that point to a specific script to
verify.

rnjudge · 2020-12-10T20:06:43Z

Do we want to put an example of how to use the --step debug subcommand in the documentation?

rnjudge · 2020-12-10T20:26:14Z

tern/analyze/default/debug/run.py

+            # cleanup
+            rootfs.clean_up()
+    else:
+        print("Something when wrong in loading the image")


Small nit: something went wrong.

nishakm · 2020-12-10T20:35:10Z

Do we want to put an example of how to use the --step debug subcommand in the documentation?

Sure, I can include that with the final commit. Should it go in CONTRIBUTING.md since it's a tern development specific feature?

rnjudge · 2020-12-10T21:37:09Z

I got an error when testing the --keys option on a golang image with lots of layers. I did not hit this with a single-layer photon image.

# tern debug -i goabhay:latest --keys base go names --shell "/usr/bin/sh"
.
.
.
Traceback (most recent call last):
  File "/home/rjudge/tt/bin/tern", line 10, in <module>
    sys.exit(main())
  File "/home/rjudge/tt/nisha/tern/tern/__main__.py", line 234, in main
    do_main(args)
  File "/home/rjudge/tt/nisha/tern/tern/__main__.py", line 98, in do_main
    derun.execute_debug(args)
  File "/home/rjudge/tt/nisha/tern/tern/analyze/default/debug/run.py", line 188, in execute_debug
    execute_invoke(full_image, args)
  File "/home/rjudge/tt/nisha/tern/tern/analyze/default/debug/run.py", line 126, in execute_invoke
    mount_container_image(image_obj, args.driver)
  File "/home/rjudge/tt/nisha/tern/tern/analyze/default/debug/run.py", line 44, in mount_container_image
    image_obj, len(image_obj.layers), driver)
  File "/home/rjudge/tt/nisha/tern/tern/analyze/default/container/multi_layer.py", line 32, in mount_overlay_fs
    tar_layers.append(image_obj.layers[index].tar_file)
IndexError: list index out of range

When running the keys option on a photon single-layer image, the debug output information is somewhat hidden in the terminal output like so:

2020-12-10 13:08:07,419 - DEBUG - rootfs - Running command: sudo mount -o bind /dev /home/rjudge/.tern/temp/mergedir/dev
2020-12-10 13:08:07,474 - DEBUG - rootfs - Running command: sudo cp /etc/resolv.conf /home/rjudge/.tern/temp/mergedir/etc/resolv.conf
2020-12-10 13:08:07,499 - DEBUG - rootfs - Running command: sudo unshare -pf --mount-proc=/home/rjudge/.tern/temp/mergedir/proc chroot /home/rjudge/.tern/temp/mergedir /usr/bin/sh -c tdnf check-update > /dev/null && tdnf list installed | cut -f1 -d"."
Output list: bash bzip2-libs ca-certificates ca-certificates-pki curl curl-libs e2fsprogs-libs elfutils-libelf expat expat-libs filesystem glibc krb5 libcap libdb libgcc libmetalink libsolv libssh2 lua ncurses-libs nspr nss-libs openssl photon-release photon-repos popt readline rpm-libs sqlite-libs tdnf tdnf-cli-libs toybox xz-libs zlib
Error messages: 
Number of elements: 35
2020-12-10 13:08:58,477 - DEBUG - rootfs - Running command: sudo umount /home/rjudge/.tern/temp/mergedir/proc
2020-12-10 13:08:58,519 - DEBUG - rootfs - Running command: sudo umount /home/rjudge/.tern/temp/mergedir/sys

do we want to find a way to separate the debug output from the terminal activity output? This might be out of scope for this PR, and if so I can open another issue for it.

The step option is nice! Only one small nit that I think needs to change for clarity. When I run tern debug -i goabhay:latest --step, I am told to pick a layer, so I do, and the directions I get are:

Run 'cd /home/rjudge/.tern/temp/mergedir && sudo chroot . /bin/sh' to look around
Run 'cd -' to go back and 'tern debug --recover' to clean up

However, cd - without first exiting the chroot environment won't work. So there should be a step to first run exit prior to running cd -.

I hit the following error when running tern debug --recover after doing a debug operation (i.e. step or keys)on an image that has multiple layers. If I run a debug operation on a single later image, the --recover option works properly for me.

2020-12-10 13:04:05,983 - DEBUG - __main__ - Starting...
2020-12-10 13:04:05,984 - DEBUG - prep - Setting up...
2020-12-10 13:04:06,127 - DEBUG - rootfs - Running command: sudo umount /home/rjudge/.tern/temp/mergedir/proc
2020-12-10 13:04:06,170 - DEBUG - rootfs - Running command: sudo umount /home/rjudge/.tern/temp/mergedir/sys
2020-12-10 13:04:06,214 - DEBUG - rootfs - Running command: sudo umount /home/rjudge/.tern/temp/mergedir/dev
2020-12-10 13:04:06,261 - DEBUG - rootfs - Running command: sudo umount -rl /home/rjudge/.tern/temp/mergedir
2020-12-10 13:04:06,321 - DEBUG - rootfs - Running command: sudo rm -rf /home/rjudge/.tern/temp/mergedir
2020-12-10 13:04:06,347 - DEBUG - rootfs - Running command: sudo rm -rf /home/rjudge/.tern/temp/workdir
2020-12-10 13:04:06,373 - DEBUG - prep - Tearing down...
Traceback (most recent call last):
  File "/home/rjudge/tt/bin/tern", line 10, in <module>
    sys.exit(main())
  File "/home/rjudge/tt/nisha/tern/tern/__main__.py", line 234, in main
    do_main(args)
  File "/home/rjudge/tt/nisha/tern/tern/__main__.py", line 100, in do_main
    prep.teardown(args.keep_wd)
  File "/home/rjudge/tt/nisha/tern/tern/prep.py", line 44, in teardown
    clean_working_dir()
  File "/home/rjudge/tt/nisha/tern/tern/prep.py", line 62, in clean_working_dir
    shutil.rmtree(path)
  File "/usr/lib/python3.6/shutil.py", line 486, in rmtree
    _rmtree_safe_fd(fd, path, onerror)
  File "/usr/lib/python3.6/shutil.py", line 424, in _rmtree_safe_fd
    _rmtree_safe_fd(dirfd, fullname, onerror)
  File "/usr/lib/python3.6/shutil.py", line 424, in _rmtree_safe_fd
    _rmtree_safe_fd(dirfd, fullname, onerror)
  File "/usr/lib/python3.6/shutil.py", line 424, in _rmtree_safe_fd
    _rmtree_safe_fd(dirfd, fullname, onerror)
  [Previous line repeated 4 more times]
  File "/usr/lib/python3.6/shutil.py", line 444, in _rmtree_safe_fd
    onerror(os.unlink, fullname, sys.exc_info())
  File "/usr/lib/python3.6/shutil.py", line 442, in _rmtree_safe_fd
    os.unlink(name, dir_fd=topfd)
PermissionError: [Errno 13] Permission denied: 'TODO'

nishakm · 2020-12-11T18:01:15Z

Do we want to put an example of how to use the --step debug subcommand in the documentation?

Sure, I can include that with the final commit. Should it go in CONTRIBUTING.md since it's a tern development specific feature?

@rnjudge any feedback on this?

rnjudge · 2020-12-11T18:07:12Z

Do we want to put an example of how to use the --step debug subcommand in the documentation?

Sure, I can include that with the final commit. Should it go in CONTRIBUTING.md since it's a tern development specific feature?

@nishakm I think it might be better as a subsection in adding-to-the-command-library.md? The other debug functionality is there and the step functionality is helpful if someone is trying to add to the command library but not getting the output they expect from the debug --keys output. What do you think?

nishakm · 2020-12-11T18:13:35Z

Do we want to put an example of how to use the --step debug subcommand in the documentation?

Sure, I can include that with the final commit. Should it go in CONTRIBUTING.md since it's a tern development specific feature?

@nishakm I think it might be better as a subsection in adding-to-the-command-library.md? The other debug functionality is there and the step functionality is helpful if someone is trying to add to the command library but not getting the output they expect from the debug --keys output. What do you think?

I will put some documentation in there for now.

nishakm · 2020-12-11T21:54:31Z

1. When running the keys option on a photon single-layer image, the debug output information is somewhat hidden in the terminal output like so:

2020-12-10 13:08:07,419 - DEBUG - rootfs - Running command: sudo mount -o bind /dev /home/rjudge/.tern/temp/mergedir/dev
2020-12-10 13:08:07,474 - DEBUG - rootfs - Running command: sudo cp /etc/resolv.conf /home/rjudge/.tern/temp/mergedir/etc/resolv.conf
2020-12-10 13:08:07,499 - DEBUG - rootfs - Running command: sudo unshare -pf --mount-proc=/home/rjudge/.tern/temp/mergedir/proc chroot /home/rjudge/.tern/temp/mergedir /usr/bin/sh -c tdnf check-update > /dev/null && tdnf list installed | cut -f1 -d"."
Output list: bash bzip2-libs ca-certificates ca-certificates-pki curl curl-libs e2fsprogs-libs elfutils-libelf expat expat-libs filesystem glibc krb5 libcap libdb libgcc libmetalink libsolv libssh2 lua ncurses-libs nspr nss-libs openssl photon-release photon-repos popt readline rpm-libs sqlite-libs tdnf tdnf-cli-libs toybox xz-libs zlib
Error messages: 
Number of elements: 35
2020-12-10 13:08:58,477 - DEBUG - rootfs - Running command: sudo umount /home/rjudge/.tern/temp/mergedir/proc
2020-12-10 13:08:58,519 - DEBUG - rootfs - Running command: sudo umount /home/rjudge/.tern/temp/mergedir/sys

do we want to find a way to separate the debug output from the terminal activity output? This might be out of scope for this PR, and if so I can open another issue for it.

I've made it look like this:

2020-12-11 13:48:30,831 - DEBUG - rootfs - Running command: sudo unshare -pf --mount-proc=/home/nisha/.tern/temp/mergedir/proc chroot /home/nisha/.tern/temp/mergedir /usr/bin/sh -c go list -m all | tail -n +2 | cut -d ' ' -f 1
2020-12-11 13:48:30,937 - ERROR - rootfs - Command failed. go list -m: not using modules

2020-12-11 13:48:30,938 - WARNING - rootfs - Error executing command in chroot
2020-12-11 13:48:30,938 - WARNING - collect - Error executing snippets: Command 'go list -m all | tail -n +2 | cut -d ' ' -f 1' returned non-zero exit status 1.

*************************************************************
          Command Library Script Verification mode           
*************************************************************

Output list: 
Error messages: go list -m: not using modules

Number of elements: 0

2020-12-11 13:48:30,939 - DEBUG - rootfs - Running command: sudo umount /home/nisha/.tern/temp/mergedir/proc
...

1. The step option is nice! Only one small nit that I think needs to change for clarity. When I run `tern debug -i goabhay:latest --step`, I am told to pick a layer, so I do, and the directions I get are:

Run 'cd /home/rjudge/.tern/temp/mergedir && sudo chroot . /bin/sh' to look around
Run 'cd -' to go back and 'tern debug --recover' to clean up

However, cd - without first exiting the chroot environment won't work. So there should be a step to first run exit prior to running cd -.

I've changed the messages this way:

Run 'cd /home/nisha/.tern/temp/mergedir && sudo chroot . /bin/sh' to look around

After exiting from your session, run 'cd -' to go back and 'tern debug --recover' to clean up.

This is work towards tern-tools#660 "Debug" is a loose term with regards to this kind of operation. We're basically "stepping-though" the layers of the container image or running scripts in an already mounted environment. This subcommand covers operations that will be useful for developers of Tern to troubleshoot any issues with the default operation. The entrypoint is in tern/analyze/default/debug/run.py. This is does not use a logger and is meant to be used in a development environment where stdout is available to look at. We move verify_invoke.py into a folder called "debug" under the default operation where "debug" will contain all the operations related to stepping through a container image. We then edit run.py to use the existing module paths. Finally, we add the subcommand "debug" to __main__.py and check if the user has selected it in do_main. Signed-off-by: Nisha K <nishak@vmware.com>

This is work towards tern-tools#660 This commit moves the container_debug.py script into the debug subcommand. We add a '--step' option to go into interactive mode where we ask what layer the user wants to inspect. When the user chooses the layer, Tern will mount upto the layer selected and provides a command to chroot into the layer to look around. We also roll in the '--recover' option to unmount the layers after the user is done inspecting. We can now remove the container_debug.py script Signed-off-by: Nisha K <nishak@vmware.com>

This completes work towards tern-tools#660 - Updated the instructions to verify scripts in base.yml using 'tern debug'. - Removed instructions to verify scripts in snippets.yml as all of them just point to scripts in base.yml - Added instructions to use 'tern debug --step' to inspect a container image at a given layer Signed-off-by: Nisha K <nishak@vmware.com>

rnjudge

The errors I was seeing are gone. This looks good! Merging now.

nishakm requested a review from rnjudge December 8, 2020 00:03

rnjudge reviewed Dec 10, 2020

View reviewed changes

Nisha K added 3 commits December 11, 2020 14:57

nishakm force-pushed the 660-step-subcommand branch from da8bd15 to f09fa05 Compare December 11, 2020 23:25

nishakm requested a review from rnjudge December 11, 2020 23:27

rnjudge approved these changes Dec 14, 2020

View reviewed changes

rnjudge merged commit 4107298 into tern-tools:master Dec 14, 2020

nishakm deleted the 660-step-subcommand branch March 25, 2021 13:43

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Introduce debug subcommand#842

Introduce debug subcommand#842
rnjudge merged 3 commits intotern-tools:masterfrom
nishakm:660-step-subcommand

nishakm commented Dec 7, 2020

Uh oh!

rnjudge commented Dec 10, 2020

Uh oh!

rnjudge Dec 10, 2020

Uh oh!

nishakm commented Dec 10, 2020

Uh oh!

rnjudge commented Dec 10, 2020

Uh oh!

nishakm commented Dec 11, 2020

Uh oh!

rnjudge commented Dec 11, 2020

Uh oh!

nishakm commented Dec 11, 2020

Uh oh!

nishakm commented Dec 11, 2020

Uh oh!

rnjudge left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

nishakm commented Dec 7, 2020

Uh oh!

rnjudge commented Dec 10, 2020

Uh oh!

rnjudge Dec 10, 2020

Choose a reason for hiding this comment

Uh oh!

nishakm commented Dec 10, 2020

Uh oh!

rnjudge commented Dec 10, 2020

Uh oh!

nishakm commented Dec 11, 2020

Uh oh!

rnjudge commented Dec 11, 2020

Uh oh!

nishakm commented Dec 11, 2020

Uh oh!

nishakm commented Dec 11, 2020

Uh oh!

rnjudge left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants