Add atomic scanner for pip, npm and gem package managers #130

dharmit · 2016-12-22T12:41:43Z

No description provided.

dharmit · 2016-12-23T11:36:18Z

#dotests

dharmit · 2016-12-28T15:42:39Z

In this PR, I've added a new beanstalk_worker/scanner.py which defines a class Scanner. It acts as a base class and can be used by the classes for individual scanners inside beanstalk_worker/worker_start_scan.py. An example of this is the MiscPackageUpdates class which uses the methods of base class to perform atomic scan.

@navidshaikh We can modify ScannerRPMVerify class to use the same as base class and remove duplicate code.

@bamachrn @mohammedzee1000 @navidshaikh need PR review.

bamachrn

LGTM

bamachrn · 2016-12-30T10:40:19Z

atomic_scanners/misc-package-updates/misc-package-updates

@@ -0,0 +1,20 @@
+type: scanner
+scanner_name: misc-package-updates
+image_name: registry.centos.org/pipeline-images/misc-package-updates


we are meaning it would be built with :latest tag right?

@bamachrn yes, it'll be built with latest tag.

bamachrn · 2016-12-30T12:20:16Z

beanstalk_worker/scanner.py

@@ -0,0 +1,95 @@
+#!/usr/bin/env python


this file is the central scanner right?

@bamachrn yep, it's the base class that would be inherited by other scanners.

navidshaikh · 2017-01-04T09:09:15Z

atomic_scanners/misc-package-updates/README.md

+--------------------------------
+
+This is a container image scanner based on `atomic scan`. The goal of the
+scanner is to scan CentOS based Docker images in the CentOS Community Container


*CentOS based container images

navidshaikh · 2017-01-04T09:09:40Z

atomic_scanners/misc-package-updates/README.md

+
+Steps to use:
+
+- Pull Docker image from **registry.centos.org**:


*Pull container image from registry.centos.org

navidshaikh · 2017-01-04T09:12:52Z

atomic_scanners/misc-package-updates/scanner.py

+    return response
+
+
+def list_of_outdated_packages(response):


Updated. Not sure why GitHub doesn't say "Show outdated".

navidshaikh · 2017-01-04T09:15:56Z

atomic_scanners/misc-package-updates/scanner.py

+
+    # remove the container
+    client.remove_container(container=container.get("Id"), force=True)
+except Exception as e:


log the exception /error message and we should put another error message about failure.

Updated. Not sure why GitHub doesn't say "Show outdated".

navidshaikh · 2017-01-04T09:18:46Z

provisions/roles/scanner/tasks/main.yml

+- name: Set SELinux to permissive
+  selinux:
+      state: permissive
+


why is this needed?

@navidshaikh because we're mounting Docker socket /var/run/docker.sock inside the container that's going to scan the image. Using which a new container is spunn off to check for respective package manager updates.

msrb · 2017-01-05T09:08:45Z

atomic_scanners/misc-package-updates/scanner.py

+    elif cli_arg == "npm":
+        exe = client.exec_create(
+            container=container.get("Id"),
+            cmd="npm outdated"


I am not sure if this command can do the trick. There can be modules installed globally (npm list -g) and also in npm_modules directories located basically anywhere on the filesystem.

@msrb agreed. This one takes care of globally installed npm modules. However, I doubt we can go through all the directories on the filesystem.

Also, what you mention about npm holds true for pip as well. If a dev prefers to use virtualenv inside the container, we are going to face similar issue.

It boils down to whether a dev wants to use npm/pip inside container in such a way that they can have multiple environments or a global environment. Our scanner should handle the latter scenario. Not sure if we can handle the former effectively.

Although ideally, putting stuff into containers should eliminate the need of again using a virtualenv as the container itself is a box in its own right.

The same should apply to npm.

@dharmit I am no NPM expert, so don't quote me on this :) I think if you want to check globally installed modules, you need to add -g, i.e.: npm outdated -g. Otherwise you're checking modules installed in ./npm_modules/ directory.

@msrb gotcha. I'm not an npm expert either. @bamachrn any thoughts?

@mohammedzee1000 whatever path you guys will take, just make sure that known limitations are documented. And you'll be fine 😉

@msrb I hate to admit it but I hadn't looked at the npm_modules dir that gets created whenever you do a npm install <package>@<version>. The -g tip is more than helpful. It's savior because that's what I actually wanted to accomplish. Having worked with pip, I didn't realize npm would create a dir at whatever location you execute npm install if it doesn't have a -g switch.

Thank you!

navidshaikh · 2017-01-06T12:02:26Z

#dotests

navidshaikh

LGTM

dharmit · 2017-01-09T03:22:20Z

#dotests

dharmit · 2017-01-11T10:18:42Z

#dotests

dharmit force-pushed the misc-package-updates branch 4 times, most recently from 64eb0c3 to 3f9127e Compare December 23, 2016 11:20

dharmit force-pushed the misc-package-updates branch 20 times, most recently from 181bd19 to 6147a21 Compare December 28, 2016 14:09

dharmit changed the title ~~[WIP] Add atomic scanner for pip, npm and gem package managers~~ Add atomic scanner for pip, npm and gem package managers Dec 28, 2016

dharmit force-pushed the misc-package-updates branch from 6147a21 to 8356008 Compare December 28, 2016 15:48

bamachrn approved these changes Dec 30, 2016

View reviewed changes

navidshaikh requested changes Jan 4, 2017

View reviewed changes

dharmit force-pushed the misc-package-updates branch from 8356008 to 03099bc Compare January 4, 2017 10:06

msrb reviewed Jan 5, 2017

View reviewed changes

dharmit force-pushed the misc-package-updates branch 9 times, most recently from e6fdb36 to b9979de Compare January 6, 2017 08:34

Add atomic scanner for pip, npm and gem package managers

44c8c99

dharmit force-pushed the misc-package-updates branch from b9979de to 44c8c99 Compare January 6, 2017 10:14

navidshaikh approved these changes Jan 6, 2017

View reviewed changes

dharmit mentioned this pull request Jan 9, 2017

Adds atomic scanner for pip, npm and gem to index CentOS/container-index#90

Merged

navidshaikh mentioned this pull request Jan 9, 2017

Logs UI #125

Merged

bamachrn merged commit f5e4526 into CentOS:master Jan 12, 2017

dharmit deleted the misc-package-updates branch January 13, 2017 14:59

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add atomic scanner for pip, npm and gem package managers #130

Add atomic scanner for pip, npm and gem package managers #130

dharmit commented Dec 22, 2016

dharmit commented Dec 23, 2016

dharmit commented Dec 28, 2016

bamachrn left a comment

bamachrn Dec 30, 2016

dharmit Jan 4, 2017

bamachrn Dec 30, 2016

dharmit Jan 4, 2017

navidshaikh Jan 4, 2017

navidshaikh Jan 4, 2017

navidshaikh Jan 4, 2017

dharmit Jan 5, 2017

navidshaikh Jan 4, 2017

dharmit Jan 5, 2017

navidshaikh Jan 4, 2017

dharmit Jan 4, 2017

msrb Jan 5, 2017

dharmit Jan 5, 2017

mohammedzee1000 Jan 5, 2017

msrb Jan 5, 2017

dharmit Jan 5, 2017

msrb Jan 5, 2017

dharmit Jan 6, 2017

navidshaikh commented Jan 6, 2017

navidshaikh left a comment

dharmit commented Jan 9, 2017

dharmit commented Jan 11, 2017


		Steps to use:

		- Pull Docker image from registry.centos.org:

Add atomic scanner for pip, npm and gem package managers #130

Add atomic scanner for pip, npm and gem package managers #130

Conversation

dharmit commented Dec 22, 2016

dharmit commented Dec 23, 2016

dharmit commented Dec 28, 2016

bamachrn left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

navidshaikh commented Jan 6, 2017

navidshaikh left a comment

Choose a reason for hiding this comment

dharmit commented Jan 9, 2017

dharmit commented Jan 11, 2017