Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(server): support server mode #678

Merged
merged 7 commits into from Jul 10, 2018

Conversation

knqyf263
Copy link
Contributor

@knqyf263 knqyf263 commented Jul 5, 2018

What did you implement:

Support server mode

How did you implement it:

Add server subcommand.

Support application/json and text/plain.
In the case of text/plain, X-Vuls-OS-Family, X-Vuls-OS-Release and X-Vuls-Kernel-Release headers are required.

Support OS: RHEL/CentOS/Debian/CentOS

How can we verify it:

Prepare Vuls server

$ vuls server -listen 0.0.0.0:5515
[Jul  5 22:32:46]  INFO [localhost] Validating config...
...
[Jul  5 22:32:46]  INFO [localhost] Listening on 0.0.0.0:5515

CentOS

text/plain

CentOS 6

$ ssh vuls@centos6
$ curl -X POST -H "Content-Type: text/plain" -H "X-Vuls-OS-Family: `awk '{print tolower($1)}' /etc/redhat-release`" -H "X-Vuls-OS-Release: `awk '{print $3}' /etc/r
edhat-release`" -H "X-Vuls-Kernel-Release: `uname -r`" --data-binary "`rpm -qa --queryformat "%{NAME} %{EPOCHNUM} %{VERSION} %{RELEASE} %{ARCH}\n"`" http://192.168.33.1:5515/

application/json

$ cat centos6.json
{
  "Family": "centos",
  "Release": "6.9",
  "RunningKernel": {
    "Release": "2.6.32-696.6.3.el6.x86_64",
    "Version": "",
    "RebootRequired": false
  },
  "Packages": {
    "ntp": {
      "Name": "ntp",
      "Version": "4.2.6p5"
    }
  }
}
$ curl -X POST -H "Content-Type: application/json" -d @centos.json http://localhost:5515

Debian

text/plain

When Debian, X-Vuls-Kernel-Release is also required.

$ ssh vuls@debian8
$ curl -X POST -H "Content-Type: text/plain" -H "X-Vuls-OS-Family: debian" -H "X-Vuls-OS-Release: 8.10" -H "X-Vuls-Kernel-Release: `uname -r`" -H "X-Vuls-Kernel-Version: `uname -a | awk '{print $7}'`" --data-binary "$(dpkg-query -W -f="\${binary:Package},\${db:Status-Abbrev},\${Version},\${Source},\${source:Version}\n")" http://192.168.33.1:5515/

application/json

$ cat debian8.json
{
  "Family": "debian",
  "Release": "8.10",
  "RunningKernel": {
    "Release": "3.16.0-4-amd64",
    "Version": "3.16.51-2",
    "RebootRequired": false
  },
  "Packages": {
    "bind9-host": {
      "Name": "bind9-host",
      "Version": "1:9.9.5.dfsg-9+deb8u15"
    }
  },
  "SrcPackages": {
    "bind9": {
      "Name": "bind9",
      "Version": "1:9.9.5.dfsg-9+deb8u15",
      "BinaryNames": [
        "bind9-host"
      ]
    }
  }
}
$ curl -X POST -H "Content-Type: application/json" -d @debian8.json http://localhost:5515

Ubuntu

text/plain

$ curl -X POST -H "Content-Type: text/plain" -H "X-Vuls-OS-Family: ubuntu" -H "X-Vuls-OS-Release: 16.04" -H "X-Vuls-Kernel-Release: `uname -r`" --data-binary "$(dpkg-query -W -f="\${binary:Package},\${db:Status-Abbrev},\${Version},\${Source},\${source:Version}\n")" http://192.168.33.1:5515/

application/json

Almost the same as Debian.
No need for RunningKernel.Version.

$ cat ubuntu1604.json
{
  "Family": "ubuntu",
  "Release": "16.04",
  "RunningKernel": {
    "Release": "3.16.0-4-amd64",
    "RebootRequired": false
  },
  "Packages": {
    "bind9-host": {
      "Name": "bind9-host",
      "Version": "1:9.9.5.dfsg-9+deb8u15"
    }
  },
  "SrcPackages": {
    "bind9": {
      "Name": "bind9",
      "Version": "1:9.9.5.dfsg-9+deb8u15",
      "BinaryNames": [
        "bind9-host"
      ]
    }
  }
}
$ curl -X POST -H "Content-Type: application/json" http://localhost:5515 -d @ubuntu1604.json

Todos:

You don't have to satisfy all of the following.

  • Write tests
  • Write documentation
  • Check that there aren't other open pull requests for the same issue/feature
  • Format your source code by make fmt
  • Pass the test by make test
  • Provide verification config / commands
  • Enable "Allow edits from maintainers" for this PR
  • Update the messages below

Is this ready for review?: NO
Is it a breaking change?: NO

@knqyf263 knqyf263 self-assigned this Jul 5, 2018
@knqyf263 knqyf263 changed the title [WIP] feat(server): support server mode feat(server): support server mode Jul 7, 2018
@kotakanbe kotakanbe merged commit eaab7a8 into future-architect:v0.5.0 Jul 10, 2018
@knqyf263 knqyf263 deleted the add_server branch July 10, 2018 06:17
kotakanbe added a commit that referenced this pull request Jul 10, 2018
* v0.5.0:
  feat(server): support server mode (#678)
  fix(report): record not found on reporting with OVAL #679 (#680)
@kotakanbe
Copy link
Member

#478

kotakanbe added a commit that referenced this pull request Aug 27, 2018
* Change config.toml, Auto-generate UUIDs, change structure of optional field

* Detect processes affected by update using yum-ps (#482)

Detect processes affected by update using yum-ps

* Detect processes needs restart using checkrestart on Debian and Ubuntu.

* pass cpename by args when calling FillCveInfo (#513)

* fix new db (#502)

* Include Version,Revision in JSON

* Include hostname in JSON

* Update goval-dictionary's commit hash in Gopkg.lock

* Remove README.ja.md

* update packages (#596)

* fix: change ControlPath to .vuls of SSH option (#618)

* feat: checkrestart for Ubuntu and Debian (#622)

* feat: checkrestart for Ubuntu and Debian

* fix: dependencies check logic of configtest

* feat: need-restarting on RedHat

* refactor: Process.ProcName to Process.Name

* feat: detect a systemd service name of need-restarting-process

* feat: detect a systemd service name of need-restarting-process on Ubuntu

* feat: fill a service name of need-restarting-process, init-system

* Support NVD JSON and CVSS3 of JVN (#605)

* fix: compile errors

* fix: Show CVSS3 on TUI

* fix: test cases

* fix: Avoid null in JSON

* Fix maxCvssScore (#621)

* Fix maxCvssScore

* Update vulninfos.go

* fix(init): remove unnecessary log initialization

* refactor(nvd): use only json feed if exists json data. if not, use xml feed

* fix(scan): make Confidence slice

* feat(CWE): Display CWE name to TUI

* feat(cwe): import CWE defs in Japanese

* feat(cwe): add OWASP Top 10 ranking to CWE if applicable

* feat(scan): add -fast-root mode, implement scan/amazon.go

* refactor(const): change const name JVN to Jvn

* feat(scan): add -fast-root mode, implement scan/centos.go

* refactor(dep): update deps

* fix(amazon): deps check

* feat(scan): add -fast-root mode, implement scan/rhel.go

* feat(scan): add -fast-root mode, implement scan/oracle.go

* fix complile err

* feat(scan): add -fast-root mode, implement scan/debian.go

* fix testcase

* fix(amazon): scan using yum

* fix(configtest): change error message, status when no scannnable servers

* Fix(scan): detect init process logic

* fix(tui): display cvss as table format

* fix(scan): parse a output of reboot-notifier on CentOS6.9

* fix(tui): don't display score, vector when score is zero

* fix(scan): add -offline mode to suse scanner

* fix(scan): fix help message

* feat(scan): enable to define scan mode for each servers in config.toml #510

* refactor(config): chagne cpeNames to cpeURIs

* refactor(config): change dependencyCheckXMLPath to owaspDCXMLPath

* fix(config): containers -> containersIncluded, Excluded, containerType

* feature(report): enable to define cpeURIs for each contaner

* feature(report): enable to specify owasp dc xml path for each container

* fix(discover): fix a template displayed at the end of discover

* feature(report): add ignorePkgsRegexp #665

* feature(report): enable to define ignoreCves for each container #666

* fix(report): Displayed nothing in TUI detail area when CweID is nil

* Gopkg.toml diet

* feat(server): support server mode (#678)

* feat(server): support server mode

* Lock go version

* Use the latest kernel release among the installed release when the running kernel release is unknown

* Add TestViaHTTP

* Set logger to go-cve-dictionary client

* Add -to-localfile

* Add -to-http option to report

* Load -to-http conf from config.toml

* Support gost (#676)

* feat(gost): Support RedHat API

* feat(gost): Support Debian Security Tracker

* feat(db): display error msg when SQLite3 is locked at the beginning of reporting.

* feat(gost): TUI

* Only use RedHat information of installed packages

* feat(tui): show mitigation on TUI

* feat(gost): support redis backend

* fix test case

* fix nil pointer when db is nil

* fix(gost): detect vulns of src packages for Debian

* feat(gost): implement redis backend for gost redhat api

* feat(report): display fixState of unfixed pkgs

* fix(report): display distincted cweIDs

* feat(slack): display gost info

* feat(slack): display mitigation

* feat(report): display available patch state as fixed/total

* fix(tui): display - if source of reference is empty

* update deps

* fix(report): key in ScanResult JSON be lowerCamelcase.

* some keys to lower camel

* fix(configtest): dep check logic of yum-plugin-ps

* fix(tui): format

* feat(report): add -format-list option

* fix(report): -format-full-text

* fix(report): report -format-full-text

* fix(report): display v3 score detected by gost

* fix(scan): scan in fast mode if not defined in config.toml

* fix(gost): fetch RedHat data for fixed CVEs

* feat(report): show number of cves detected in each database

* fix(report): show new version as `Unknown` in offline and fast scan mode

* fix(report): fix num of upadtable and fixed

* fix(report): set `Not fixed yet` if packageStatus is empty

* refact(gost): make convertToModel public

* fix(test): fix test case

* update deps

* fix(report): include gost score in MaxCvssScore

* [WIP] feat(config): enable to set options in config.toml instead of cmd opt (#690)

* feat(config): enable to set options in config.toml instead of cmd opt

* fix(config): change Conf.Report.Slack to Conf.Slack

* fix(discover): change tempalte

* fix(report): fix config.toml auto-generate with -uuid

* Add endpoint for health check and change endpoint

* refact(cmd): refactor flag set

* fix(report): enable to specify opts with cmd arg and env value

* fix(scan): enable to parse the release version of amazon linux 2

* add(report) add -to-saas option (#695)

* add(report) add -to-saas option

* ignore other writer if -to-saas

* fix(saas) fix bug

* fix(scan): need-restarting needs internet connection

* fix(scan,configtest): check scan mode

* refactor(scan): change func name

* fix(suse): support offline mode, bug fix on AWS, zypper --no-color

* fix(tui): fix nil pointer when no vulns in tui

* feat(report): enable to define CPE FS format in config.toml

* fix(vet): fix warnings of go vet

* fix(travis): go version to 1.11

* update deps
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants