Setup Documentation Unclear #96

rgpass · 2019-03-07T15:28:50Z

I am looking to experiment with NodeJsScan, but I am having trouble following the setup steps. Is it possible to add more explicit steps to getting started?

rgpass · 2019-03-07T15:57:28Z

I cloned the repo, installed python3.

pip3 install -r requirements.txt # worked fine
python3 migrate.py # failed

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 2262, in _wrap_pool_connect
    return fn()
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/pool/base.py", line 354, in connect
    return _ConnectionFairy._checkout(self)
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/pool/base.py", line 751, in _checkout
    fairy = _ConnectionRecord.checkout(pool)
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/pool/base.py", line 483, in checkout
    rec = pool._do_get()
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/pool/impl.py", line 138, in _do_get
    self._dec_overflow()
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/util/langhelpers.py", line 68, in __exit__
    compat.reraise(exc_type, exc_value, exc_tb)
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 129, in reraise
    raise value
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/pool/impl.py", line 135, in _do_get
    return self._create_connection()
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/pool/base.py", line 299, in _create_connection
    return _ConnectionRecord(self)
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/pool/base.py", line 428, in __init__
    self.__connect(first_connect_check=True)
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/pool/base.py", line 630, in __connect
    connection = pool._invoke_creator(self)
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/strategies.py", line 114, in connect
    return dialect.connect(*cargs, **cparams)
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 453, in connect
    return self.dbapi.connect(*cargs, **cparams)
  File "/usr/local/lib/python3.7/site-packages/psycopg2/__init__.py", line 130, in connect
    conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
	Is the server running on host "localhost" (::1) and accepting
	TCP/IP connections on port 5432?
could not connect to server: Connection refused
	Is the server running on host "localhost" (127.0.0.1) and accepting
	TCP/IP connections on port 5432?


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "migrate.py", line 60, in <module>
    Base.metadata.create_all(bind=engine)
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/sql/schema.py", line 4287, in create_all
    ddl.SchemaGenerator, self, checkfirst=checkfirst, tables=tables
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 2032, in _run_visitor
    with self._optional_conn_ctx_manager(connection) as conn:
  File "/usr/local/Cellar/python/3.7.2_2/Frameworks/Python.framework/Versions/3.7/lib/python3.7/contextlib.py", line 112, in __enter__
    return next(self.gen)
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 2024, in _optional_conn_ctx_manager
    with self._contextual_connect() as conn:
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 2226, in _contextual_connect
    self._wrap_pool_connect(self.pool.connect, None),
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 2266, in _wrap_pool_connect
    e, dialect, self
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1536, in _handle_dbapi_exception_noconnection
    util.raise_from_cause(sqlalchemy_exception, exc_info)
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 383, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 128, in reraise
    raise value.with_traceback(tb)
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 2262, in _wrap_pool_connect
    return fn()
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/pool/base.py", line 354, in connect
    return _ConnectionFairy._checkout(self)
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/pool/base.py", line 751, in _checkout
    fairy = _ConnectionRecord.checkout(pool)
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/pool/base.py", line 483, in checkout
    rec = pool._do_get()
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/pool/impl.py", line 138, in _do_get
    self._dec_overflow()
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/util/langhelpers.py", line 68, in __exit__
    compat.reraise(exc_type, exc_value, exc_tb)
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 129, in reraise
    raise value
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/pool/impl.py", line 135, in _do_get
    return self._create_connection()
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/pool/base.py", line 299, in _create_connection
    return _ConnectionRecord(self)
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/pool/base.py", line 428, in __init__
    self.__connect(first_connect_check=True)
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/pool/base.py", line 630, in __connect
    connection = pool._invoke_creator(self)
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/strategies.py", line 114, in connect
    return dialect.connect(*cargs, **cparams)
  File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 453, in connect
    return self.dbapi.connect(*cargs, **cparams)
  File "/usr/local/lib/python3.7/site-packages/psycopg2/__init__.py", line 130, in connect
    conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) could not connect to server: Connection refused
	Is the server running on host "localhost" (::1) and accepting
	TCP/IP connections on port 5432?
could not connect to server: Connection refused
	Is the server running on host "localhost" (127.0.0.1) and accepting
	TCP/IP connections on port 5432?

(Background on this error at: http://sqlalche.me/e/e3q8)

ajinabraham · 2019-03-07T16:59:03Z

You should read the exceptions. It clearly says it's unable to connect to the DB

psycopg2.OperationalError: could not connect to server: Connection refused
	Is the server running on host "localhost" (::1) and accepting
	TCP/IP connections on port 5432?
could not connect to server: Connection refused
	Is the server running on host "localhost" (127.0.0.1) and accepting
	TCP/IP connections on port 5432?

You might have missed this step mentioned in documentation.

Install Postgres and configure SQLALCHEMY_DATABASE_URI in core/settings.py

If you just want to experiment. Use a docker image.
Install Docker CE

docker pull opensecurity/nodejsscan
docker run -it -p 9090:9090 opensecurity/nodejsscan:latest

rgpass · 2019-03-07T17:40:51Z

Yes, I read the exceptions. "Clearly", my point is that you should add better documentation for setting that up. There are several guides on how to write proper READMEs, along with a curated list of repos with good READMEs.

ajinabraham · 2019-03-07T17:48:20Z

I am not sure I understand you. What is that you expect from the readme?
It mentions every step required for configuring and setting up the tool.

# Install Postgres and configure SQLALCHEMY_DATABASE_URI in core/settings.py
pip3 install -r requirements.txt
python3 migrate.py # Run once to create database entries required
python3 app.py # Testing Environment
gunicorn -b 0.0.0.0:9090 app:app --workers 3 --timeout 10000 # Production Environment

What more info do you need or what you think is missing?

rgpass · 2019-03-07T18:20:04Z

The README should consider those who have little to no experience and ideally have the mentality of "Don't Make Me Think." This is especially important considering the topic of security.

My suggestion would be to imagine you had a brand new computer and you were walking through the steps with a junior developer. What steps would they follow?

## Getting Started

### Pre-Reqs

* Install `python3` [using this setup guide](https://realpython.com/installing-python/)
  * Note: This will install `pip3`
  * Sanity check that it's working, run the following in bash: `which python3`
    * Good sign is something like: `/usr/local/bin/python3`
* Install PostgresQL
  * The easiest way is to Google "install postgresql <mac|windows|linux>"
  * Sanity check that it's working, run the following in bash: `psql`
    * Good sign: it changes to the Postgres prompt

### Installation

* In bash, run the following
  * Clone the repo and `cd` into the repo: `git clone <repo name> && cd NodeJsScan`
  * Install dependencies: `pip3 install -r requirements.txt`
* Create a database in Postgres: `createdb nodejsscan`
* Run the migrations to set up the database: `python3 migrate.py`
* Start the server that creates a UI: `python3 app.py`
* Navigate to `http://0.0.0.0:9090/`

### Advanced Installation

* To change the NodeJsScan database, see `core/settings.py:38`

### Basic Usage

* In the UI (`http://0.0.0.0:9090/`), compress (zip) your source files and upload
* If you don't have your own source files, use the example found at `examples/source.files.zip`

### Advanced Usage

* If you're using a transpiler or other build step, you probably have a `src` folder and a `dist` folder. You will want to upload a compressed (zipped) file of `dist`, NOT `src`
* TypeScript is not currently supported

The Basic Usage section needs some work. I'm messing around with the tool now and always getting the same results, but that's another issue that I can open.

tforster · 2019-03-07T18:26:47Z

@rgpass Instead of complaining why don't you submit a PR?

rgpass · 2019-03-07T18:31:51Z

@tforster Wasn't complaining, but I plan on submitting a PR once I get it working myself. 👍

tforster · 2019-03-07T18:38:39Z

That will be awesome. It's a good tool and if it takes a little effort it is worth it. I just started using it in the last week or so and created a CLI docker file so I can incorporate in my pipeline. @ajinabraham accepted my PR right away so anything we can do to help grow his project will benefit all of us.

ajinabraham closed this as completed Mar 7, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Setup Documentation Unclear #96

Setup Documentation Unclear #96

rgpass commented Mar 7, 2019

rgpass commented Mar 7, 2019

ajinabraham commented Mar 7, 2019

rgpass commented Mar 7, 2019

ajinabraham commented Mar 7, 2019

rgpass commented Mar 7, 2019 •

edited

Loading

tforster commented Mar 7, 2019

rgpass commented Mar 7, 2019

tforster commented Mar 7, 2019

Setup Documentation Unclear #96

Setup Documentation Unclear #96

Comments

rgpass commented Mar 7, 2019

rgpass commented Mar 7, 2019

ajinabraham commented Mar 7, 2019

rgpass commented Mar 7, 2019

ajinabraham commented Mar 7, 2019

rgpass commented Mar 7, 2019 • edited Loading

tforster commented Mar 7, 2019

rgpass commented Mar 7, 2019

tforster commented Mar 7, 2019

rgpass commented Mar 7, 2019 •

edited

Loading