Library Scan Grep Pattern Is Not Exhaustive and Leads to Confusing Errors #1055

ohsh6o · 2020-10-06T15:07:54Z

What did you do? (required. The issue will be closed when not provided.)

I am running an offline scan to look at libraries only and send the output to another vuls server later with the following command with v0.12.3.

Command and configuration file are below.

What did you expect to happen?

I expected the scan to complete without error.

What happened instead?

I get errors in the output and the return value of 1, which breaks my shell scripts and automation pipelines. This is because the grep command used when shelling out is too specific, I have multiple files from the procfs pseudo-filesystem that are not failing with Permission Denied errors, but Invalid Argument these are making their way through, as referenced in the stacktrace info that points to this error check here.

I will file a PR to propose a fix to this issue once I submit this.

$ find / -name "*package-lock.json" -o -name "*yarn.lock" -o -name "*Gemfile.lock" -o -name "*Cargo.lock" -o -name "*composer.lock" -o -name "*Pipfile.lock" -o -name "*poetry.lock"  2>&1 | grep Invalid
find: ‘/proc/2155/task/2155/net’: Invalid argument
find: ‘/proc/2155/net’: Invalid argument

Current Output

Please re-run the command using -debug and provide the output below.

$ vuls scan -debug -config=config.toml -libs-only -results-dir=/home/appuser/tmp/
[Oct  6 15:00:05]  INFO [localhost] Start scanning
[Oct  6 15:00:05]  INFO [localhost] config: config.toml
[Oct  6 15:00:05] DEBUG [localhost] map[string]config.ServerInfo{}
[Oct  6 15:00:05]  INFO [localhost] Validating config...
[Oct  6 15:00:05]  INFO [localhost] Detecting Server/Container OS... 
[Oct  6 15:00:05]  INFO [localhost] Detecting OS of servers... 
[Oct  6 15:00:05] DEBUG [localhost] Executing... ls /etc/debian_version
[Oct  6 15:00:05] DEBUG [localhost] execResult: servername: 
  cmd: ls /etc/debian_version
  exitstatus: 0
  stdout: /etc/debian_version

  stderr: 
  err: %!s(<nil>)
[Oct  6 15:00:05] DEBUG [localhost] Executing... cat /etc/issue
[Oct  6 15:00:05] DEBUG [localhost] execResult: servername: 
  cmd: cat /etc/issue
  exitstatus: 0
  stdout: Ubuntu 18.04.5 LTS \n \l


  stderr: 
  err: %!s(<nil>)
[Oct  6 15:00:05] DEBUG [localhost] Executing... lsb_release -ir
[Oct  6 15:00:05] DEBUG [localhost] execResult: servername: 
  cmd: lsb_release -ir
  exitstatus: 0
  stdout: Distributor ID:       Ubuntu
Release:        18.04

  stderr: 
  err: %!s(<nil>)
[Oct  6 15:00:05] DEBUG [localhost] Debian like Linux. Host: localhost:local
[Oct  6 15:00:05]  INFO [localhost] (1/1) Detected: appserver: ubuntu 18.04
[Oct  6 15:00:05]  INFO [localhost] Detecting OS of containers... 
[Oct  6 15:00:05]  INFO [localhost] Checking Scan Modes... 
[Oct  6 15:00:05]  INFO [localhost] Detecting Platforms... 
[Oct  6 15:00:05]  INFO [localhost] (1/1) appserver is running on unknown
[Oct  6 15:00:05]  INFO [localhost] Detecting IPS identifiers... 
[Oct  6 15:00:05]  INFO [localhost] (1/1) appserver has 0 IPS integration
[Oct  6 15:00:05]  INFO [localhost] Scanning vulnerabilities... 
[Oct  6 15:00:05]  INFO [localhost] Scanning vulnerable OS packages...
[Oct  6 15:00:05] DEBUG [localhost] Executing... find / -name "*yarn.lock" -o -name "*Gemfile.lock" -o -name "*Cargo.lock" -o -name "*composer.lock" -o -name "*Pipfile.lock" -o -name "*poetry.lock" -o -name "*package-lock.json"  2>&1 | grep -v "Permission denied"
[Oct  6 15:00:06] DEBUG [localhost] execResult: servername: 
  cmd: find / -name "*yarn.lock" -o -name "*Gemfile.lock" -o -name "*Cargo.lock" -o -name "*composer.lock" -o -name "*Pipfile.lock" -o -name "*poetry.lock" -o -name "*package-lock.json"  2>&1 | grep -v "Permission denied"
  exitstatus: 0
  stdout: /home/appuser/app/Pipfile.lock
find: ‘/proc/1989/task/1989/net’: Invalid argument
find: ‘/proc/1989/net’: Invalid argument

  stderr: 
  err: %!s(<nil>)
[Oct  6 15:00:06] DEBUG [localhost] Executing... cat /home/appuser/app/Pipfile.lock
[Oct  6 15:00:06] DEBUG [localhost] execResult: servername: 
  cmd: cat /home/appuser/app/Pipfile.lock
  exitstatus: 0
  stdout: {
    "_meta": {
        "hash": {
            "sha256": "554c73514355a9b0bbfdf89104ee065c46e0a9610de93b57bc3428ff6021d79f"
        },
        "pipfile-spec": 6,
        "requires": {
            "python_version": "3.8"
        },
        "sources": [
            {
                "name": "pypi",
                "url": "https://pypi.org/simple",
                "verify_ssl": true
            }
        ]
    },
    "default": {
        "aiosqlite": {
            "hashes": [
                "sha256:19b984b6702aed9f1c85c023f37296954547fc4030dae8e9d027b2a930bed78b",
                "sha256:a2884793f4dc8f2798d90e1dfecb2b56a6d479cf039f7ec52356a7fd5f3bdc57"
            ],
            "version": "==0.15.0"
        },
        "ciso8601": {
            "hashes": [
                "sha256:bdbb5b366058b1c87735603b23060962c439ac9be66f1ae91e8c7dbd7d59e262"
            ],
            "version": "==2.1.3"
        },
        "click": {
            "hashes": [
                "sha256:d2b5255c7c6349bc1bd1e59e08cd12acbbd63ce649f2588755783aa94dfb6b1a",
                "sha256:dacca89f4bfadd5de3d7489b7c8a566eee0d3676333fbb50030263894c38c0dc"
            ],
            "version": "==7.1.2"
        },
        "flask": {
            "hashes": [
                "sha256:4efa1ae2d7c9865af48986de8aeb8504bf32c7f3d6fdc9353d34b21f4b127060",
                "sha256:8a4fdd8936eba2512e9c85df320a37e694c93945b33ef33c89946a340a238557"
            ],
            "index": "pypi",
            "version": "==1.1.2"
        },
        "flask-login": {
            "hashes": [
                "sha256:6d33aef15b5bcead780acc339464aae8a6e28f13c90d8b1cf9de8b549d1c0b4b",
                "sha256:7451b5001e17837ba58945aead261ba425fdf7b4f0448777e597ddab39f4fba0"
            ],
            "index": "pypi",
            "version": "==0.5.0"
        },
        "flask-sqlalchemy": {
            "hashes": [
                "sha256:05b31d2034dd3f2a685cbbae4cfc4ed906b2a733cff7964ada450fd5e462b84e",
                "sha256:bfc7150eaf809b1c283879302f04c42791136060c6eeb12c0c6674fb1291fae5"
            ],
            "index": "pypi",
            "version": "==2.4.4"
        },
        "gunicorn": {
            "hashes": [
                "sha256:1904bb2b8a43658807108d59c3f3d56c2b6121a701161de0ddf9ad140073c626",
                "sha256:cd4a810dd51bf497552cf3f863b575dabd73d6ad6a91075b65936b151cbf4f9c"
            ],
            "index": "pypi",
            "version": "==20.0.4"
        },
        "itsdangerous": {
            "hashes": [
                "sha256:321b033d07f2a4136d3ec762eac9f16a10ccd60f53c0c91af90217ace7ba1f19",
                "sha256:b12271b2047cb23eeb98c8b5622e2e5c5e9abd9784a153e9d8ef9cb4dd09d749"
            ],
            "version": "==1.1.0"
        },
        "jinja2": {
            "hashes": [
                "sha256:89aab215427ef59c34ad58735269eb58b1a5808103067f7bb9d5836c651b3bb0",
                "sha256:f0a4641d3cf955324a89c04f3d94663aa4d638abe8f733ecd3582848e1c37035"
            ],
            "version": "==2.11.2"
        },
        "markupsafe": {
            "hashes": [
                "sha256:00bc623926325b26bb9605ae9eae8a215691f33cae5df11ca5424f06f2d1f473",
                "sha256:09027a7803a62ca78792ad89403b1b7a73a01c8cb65909cd876f7fcebd79b161",
                "sha256:09c4b7f37d6c648cb13f9230d847adf22f8171b1ccc4d5682398e77f40309235",
                "sha256:1027c282dad077d0bae18be6794e6b6b8c91d58ed8a8d89a89d59693b9131db5",
                "sha256:13d3144e1e340870b25e7b10b98d779608c02016d5184cfb9927a9f10c689f42",
                "sha256:24982cc2533820871eba85ba648cd53d8623687ff11cbb805be4ff7b4c971aff",
                "sha256:29872e92839765e546828bb7754a68c418d927cd064fd4708fab9fe9c8bb116b",
                "sha256:43a55c2930bbc139570ac2452adf3d70cdbb3cfe5912c71cdce1c2c6bbd9c5d1",
                "sha256:46c99d2de99945ec5cb54f23c8cd5689f6d7177305ebff350a58ce5f8de1669e",
                "sha256:500d4957e52ddc3351cabf489e79c91c17f6e0899158447047588650b5e69183",
                "sha256:535f6fc4d397c1563d08b88e485c3496cf5784e927af890fb3c3aac7f933ec66",
                "sha256:596510de112c685489095da617b5bcbbac7dd6384aeebeda4df6025d0256a81b",
                "sha256:62fe6c95e3ec8a7fad637b7f3d372c15ec1caa01ab47926cfdf7a75b40e0eac1",
                "sha256:6788b695d50a51edb699cb55e35487e430fa21f1ed838122d722e0ff0ac5ba15",
                "sha256:6dd73240d2af64df90aa7c4e7481e23825ea70af4b4922f8ede5b9e35f78a3b1",
                "sha256:717ba8fe3ae9cc0006d7c451f0bb265ee07739daf76355d06366154ee68d221e",
                "sha256:79855e1c5b8da654cf486b830bd42c06e8780cea587384cf6545b7d9ac013a0b",
                "sha256:7c1699dfe0cf8ff607dbdcc1e9b9af1755371f92a68f706051cc8c37d447c905",
                "sha256:88e5fcfb52ee7b911e8bb6d6aa2fd21fbecc674eadd44118a9cc3863f938e735",
                "sha256:8defac2f2ccd6805ebf65f5eeb132adcf2ab57aa11fdf4c0dd5169a004710e7d",
                "sha256:98c7086708b163d425c67c7a91bad6e466bb99d797aa64f965e9d25c12111a5e",
                "sha256:9add70b36c5666a2ed02b43b335fe19002ee5235efd4b8a89bfcf9005bebac0d",
                "sha256:9bf40443012702a1d2070043cb6291650a0841ece432556f784f004937f0f32c",
                "sha256:ade5e387d2ad0d7ebf59146cc00c8044acbd863725f887353a10df825fc8ae21",
                "sha256:b00c1de48212e4cc9603895652c5c410df699856a2853135b3967591e4beebc2",
                "sha256:b1282f8c00509d99fef04d8ba936b156d419be841854fe901d8ae224c59f0be5",
                "sha256:b2051432115498d3562c084a49bba65d97cf251f5a331c64a12ee7e04dacc51b",
                "sha256:ba59edeaa2fc6114428f1637ffff42da1e311e29382d81b339c1817d37ec93c6",
                "sha256:c8716a48d94b06bb3b2524c2b77e055fb313aeb4ea620c8dd03a105574ba704f",
                "sha256:cd5df75523866410809ca100dc9681e301e3c27567cf498077e8551b6d20e42f",
                "sha256:cdb132fc825c38e1aeec2c8aa9338310d29d337bebbd7baa06889d09a60a1fa2",
                "sha256:e249096428b3ae81b08327a63a485ad0878de3fb939049038579ac0ef61e17e7",
                "sha256:e8313f01ba26fbbe36c7be1966a7b7424942f670f38e666995b88d012765b9be"
            ],
            "version": "==1.1.1"
        },
        "pypika": {
            "hashes": [
                "sha256:abf85d7fc3da6c4213125b58ca989a1eabfcc1e9b1f5fc3f524eba5cd7a25107"
            ],
            "version": "==0.38.0"
        },
        "sqlalchemy": {
            "hashes": [
                "sha256:0942a3a0df3f6131580eddd26d99071b48cfe5aaf3eab2783076fbc5a1c1882e",
                "sha256:0ec575db1b54909750332c2e335c2bb11257883914a03bc5a3306a4488ecc772",
                "sha256:109581ccc8915001e8037b73c29590e78ce74be49ca0a3630a23831f9e3ed6c7",
                "sha256:16593fd748944726540cd20f7e83afec816c2ac96b082e26ae226e8f7e9688cf",
                "sha256:427273b08efc16a85aa2b39892817e78e3ed074fcb89b2a51c4979bae7e7ba98",
                "sha256:50c4ee32f0e1581828843267d8de35c3298e86ceecd5e9017dc45788be70a864",
                "sha256:512a85c3c8c3995cc91af3e90f38f460da5d3cade8dc3a229c8e0879037547c9",
                "sha256:57aa843b783179ab72e863512e14bdcba186641daf69e4e3a5761d705dcc35b1",
                "sha256:621f58cd921cd71ba6215c42954ffaa8a918eecd8c535d97befa1a8acad986dd",
                "sha256:6ac2558631a81b85e7fb7a44e5035347938b0a73f5fdc27a8566777d0792a6a4",
                "sha256:716754d0b5490bdcf68e1e4925edc02ac07209883314ad01a137642ddb2056f1",
                "sha256:736d41cfebedecc6f159fc4ac0769dc89528a989471dc1d378ba07d29a60ba1c",
                "sha256:8619b86cb68b185a778635be5b3e6018623c0761dde4df2f112896424aa27bd8",
                "sha256:87fad64529cde4f1914a5b9c383628e1a8f9e3930304c09cf22c2ae118a1280e",
                "sha256:89494df7f93b1836cae210c42864b292f9b31eeabca4810193761990dc689cce",
                "sha256:8cac7bb373a5f1423e28de3fd5fc8063b9c8ffe8957dc1b1a59cb90453db6da1",
                "sha256:8fd452dc3d49b3cc54483e033de6c006c304432e6f84b74d7b2c68afa2569ae5",
                "sha256:adad60eea2c4c2a1875eb6305a0b6e61a83163f8e233586a4d6a55221ef984fe",
                "sha256:c26f95e7609b821b5f08a72dab929baa0d685406b953efd7c89423a511d5c413",
                "sha256:cbe1324ef52ff26ccde2cb84b8593c8bf930069dfc06c1e616f1bfd4e47f48a3",
                "sha256:d05c4adae06bd0c7f696ae3ec8d993ed8ffcc4e11a76b1b35a5af8a099bd2284",
                "sha256:d98bc827a1293ae767c8f2f18be3bb5151fd37ddcd7da2a5f9581baeeb7a3fa1",
                "sha256:da2fb75f64792c1fc64c82313a00c728a7c301efe6a60b7a9fe35b16b4368ce7",
                "sha256:e4624d7edb2576cd72bb83636cd71c8ce544d8e272f308bd80885056972ca299",
                "sha256:e89e0d9e106f8a9180a4ca92a6adde60c58b1b0299e1b43bd5e0312f535fbf33",
                "sha256:f11c2437fb5f812d020932119ba02d9e2bc29a6eca01a055233a8b449e3e1e7d",
                "sha256:f57be5673e12763dd400fea568608700a63ce1c6bd5bdbc3cc3a2c5fdb045274",
                "sha256:fc728ece3d5c772c196fd338a99798e7efac7a04f9cb6416299a3638ee9a94cd"
            ],
            "index": "pypi",
            "version": "==1.3.18"
        },
        "tortoise-orm": {
            "hashes": [
                "sha256:e5fa256f9bac59b614d0afa9de2c8f2de0cd31bb018b0006bcd44fd8f4e0fc5b"
            ],
            "index": "pypi",
            "version": "==0.16.6"
        },
        "typing-extensions": {
            "hashes": [
                "sha256:6e95524d8a547a91e08f404ae485bbb71962de46967e1b71a0cb89af24e761c5",
                "sha256:79ee589a3caca649a9bfd2a8de4709837400dfa00b6cc81962a1e6a1815969ae",
                "sha256:f8d2bd89d25bc39dabe7d23df520442fa1d8969b82544370e03d88b5a591c392"
            ],
            "version": "==3.7.4.2"
        },
        "werkzeug": {
            "hashes": [
                "sha256:2de2a5db0baeae7b2d2664949077c2ac63fbd16d98da0ff71837f7d1dea3fd43",
                "sha256:6c80b1e5ad3665290ea39320b91e1be1e0d5f60652b964a3070216de83d2e47c"
            ],
            "version": "==1.0.1"
        }
    },
    "develop": {
        "flake8": {
            "hashes": [
                "sha256:15e351d19611c887e482fb960eae4d44845013cc142d42896e9862f775d8cf5c",
                "sha256:f04b9fcbac03b0a3e58c0ab3a0ecc462e023a9faf046d57794184028123aa208"
            ],
            "index": "pypi",
            "version": "==3.8.3"
        },
        "mccabe": {
            "hashes": [
                "sha256:ab8a6258860da4b6677da4bd2fe5dc2c659cff31b3ee4f7f5d64e79735b80d42",
                "sha256:dd8d182285a0fe56bace7f45b5e7d1a6ebcbf524e8f3bd87eb0f125271b8831f"
            ],
            "version": "==0.6.1"
        },
        "pycodestyle": {
            "hashes": [
                "sha256:2295e7b2f6b5bd100585ebcb1f616591b652db8a741695b3d8f5d28bdc934367",
                "sha256:c58a7d2815e0e8d7972bf1803331fb0152f867bd89adf8a01dfd55085434192e"
            ],
            "version": "==2.6.0"
        },
        "pyflakes": {
            "hashes": [
                "sha256:0d94e0e05a19e57a99444b6ddcf9a6eb2e5c68d3ca1e98e90707af8152c90a92",
                "sha256:35b2d75ee967ea93b55750aa9edbbf72813e06a66ba54438df2cfac9e3c27fc8"
            ],
            "version": "==2.2.0"
        }
    }
}

  stderr: 
  err: %!s(<nil>)
[Oct  6 15:00:06] DEBUG [localhost] Executing... cat find: ‘/proc/1989/task/1989/net’: Invalid argument
[Oct  6 15:00:06] DEBUG [localhost] execResult: servername: 
  cmd: cat find: ‘/proc/1989/task/1989/net’: Invalid argument
  exitstatus: 1
  stdout: 
  stderr: cat: 'find:': No such file or directory
cat: '‘/proc/1989/task/1989/net’:': No such file or directory
cat: Invalid: No such file or directory
cat: argument: No such file or directory

  err: exit status 1
[Oct  6 15:00:06] ERROR [localhost] Error on appserver, err: [Failed to scan Library:
    github.com/future-architect/vuls/scan.GetScanResults.func1
        /home/runner/work/vuls/vuls/scan/serverapi.go:635
  - Failed to get target file: execResult: servername: 
      cmd: cat find: ‘/proc/1989/task/1989/net’: Invalid argument
      exitstatus: 1
      stdout: 
      stderr: cat: 'find:': No such file or directory
    cat: '‘/proc/1989/task/1989/net’:': No such file or directory
    cat: Invalid: No such file or directory
    cat: argument: No such file or directory
    
      err: exit status 1, filepath: find: ‘/proc/1989/task/1989/net’: Invalid argument:
    github.com/future-architect/vuls/scan.(*base).scanLibraries
        /home/runner/work/vuls/vuls/scan/base.go:578]


One Line Summary
================
appserver Error           Use configtest subcommand or scan with --debug to view the details

Steps to reproduce the behaviour

Run a scan command running with the following configuration.

Configuration (MUST fill this out):

Vuls environment:

To check the commit hash of HEAD
$ vuls -v

$ vuls -v
vuls 0.9.9

NOTE: This is broken, I will be looking into this and filing another issue. I use a configuration management tool to download tagged releases from Github, this version number is incorrectly coded in v0.12.3.

or

$ cd $GOPATH/src/github.com/future-architect/vuls
$ git rev-parse --short HEAD

config.toml:

[servers]

[servers.appservername]
host = "localhost"
port = "local"
findLock = true
scanMode     = ["offline"]

command:

vuls scan -debug -config=config.toml -libs-only -results-dir=/home/appuser/tmp/

The text was updated successfully, but these errors were encountered:

Fixes future-architect#1055. When find throws an error, it prefixes an error with the utility name, (`find` by default), the quoted path where find encountered the error, and the specific error message. As it stands now, vuls only filters out the 'Permission Denied' message. Informal testing in a Ubuntu system found other errors from the procfs pseudo-filesystem. So this fix will filter out all error messages, as find is only used to lock for language runtime dependency lockfiles, and nothing else.

ohsh6o added the bug label Oct 6, 2020

This was referenced Oct 6, 2020

Expand negative grep match for any error for lib scans. #1056

Merged

Version Mismatches as Version is Hard-Coded and Not Updated #1057

Closed

Expand negative grep match for any error for lib scans. 18F/vuls#2

Merged

kotakanbe closed this as completed in #1056 Oct 12, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Library Scan Grep Pattern Is Not Exhaustive and Leads to Confusing Errors #1055

Library Scan Grep Pattern Is Not Exhaustive and Leads to Confusing Errors #1055

ohsh6o commented Oct 6, 2020 •

edited

Library Scan Grep Pattern Is Not Exhaustive and Leads to Confusing Errors #1055

Library Scan Grep Pattern Is Not Exhaustive and Leads to Confusing Errors #1055

Comments

ohsh6o commented Oct 6, 2020 • edited

What did you do? (required. The issue will be closed when not provided.)

What did you expect to happen?

What happened instead?

Steps to reproduce the behaviour

Configuration (MUST fill this out):

ohsh6o commented Oct 6, 2020 •

edited