Migrates SSH Observatory datastore from MongoDB to Postgres #121

claudijd · 2018-01-09T19:30:12Z

This is a work in progress to put together a postgres backend replacement for mongodb. It's far from done, but a PR is the best place to share with others and get their feedback.

claudijd · 2018-01-09T19:39:28Z

lib/ssh_scan_api/database/postgres.rb

+
+      # Creates a database
+      def create(name="ssh_scan")
+        @client.exec("CREATE DATABASE #{name}")


All these SQL calls need to be parameterized for safety

claudijd · 2018-01-09T19:43:16Z

lib/ssh_scan_api/database/postgres.rb

+        #   "scan" => nil,
+        #   "queue_time" => Time.now,
+        #   "worker_id" => nil,
+        # )


All these interfaces need populating/porting into postgres capable

…rfaces

claudijd · 2018-01-11T16:50:45Z

With the latest push, this should start passing again. I will also be doing some bench sanity checks to make sure it still works end to end before landing this change.

claudijd · 2018-01-12T17:51:05Z

Ok, so finally able to task scans properly, so we're getting there, next up, troubleshooting the minor changes we made but forgot to adjust the interfaces for.

caggle · 2018-01-17T00:02:37Z

Instructions worked great for me, no issues with or without jq.

I did notice though, when you scan a system with 22/tcp is closed, currently the scan goes on forever? And worker instance borks some time when this happens.

SCMGuruLLC · 2018-01-17T01:07:55Z

I'm running Ubuntu Server 16.04.3 LTS 32-bit.

I ran:
3.) docker-compose up --build

And got:
ERROR: Couldn't connect to Docker daemon at http+docker://localunixsocket - is it running?
If it's at a non-standard location, specify the URL with the DOCKER_HOST environment variable.

I ran:
sudo docker-compose up --build

And got:
Creating network "sshscanapi_default" with the default driver
Building base
Step 1/8 : FROM ruby:latest AS base
ERROR: Service 'base' failed to build: Error parsing reference: "ruby:latest AS base" is not a valid repository/tag: invalid reference format

claudijd · 2018-01-17T19:02:01Z

@cag I believe ssh_scan has logic to handle the socket timeout and will drop the scan result into an error state. Unfortunately, the clients are very unsophisticated, mainly because they are for demo purposes and don't have logic to capture that error condition. That said, you should be able to verify this by accessing this route after attempting the scan...

https://sshscan.rubidus.com/api/v1/scan/results?uuid=db49ba74-8482-4c30-9cbe-66f4ae62683b

You just need to replace the UUID in the url to the UUID the client says it got for that scan. We will then return everything we have for that scan, but if the socket timed out, it will be very little, but it should be there. In other words, I believe this is working as expected, please let me know if it doesn't behave this way for you.

claudijd · 2018-01-17T19:06:31Z

@scmaccal the sudo/non-sudo behavior is simply a function of your current user not having docker privs. If you add that user to the docker group, I think that will solve the problem.

As for the second issue, I'm wondering if it's a docker version issue, would you mind sharing the output of the following:

docker-compose version
docker version

I'm assuming your version of docker doesn't support the AS capability, but I'd like to compare to be sure.

SCMGuruLLC · 2018-01-17T19:30:03Z

sudo docker-compose version docker-compose version 1.8.0, build unknown docker-py version: 1.9.0 CPython version: 2.7.12 OpenSSL version: OpenSSL 1.0.2g 1 Mar 2016 sudo docker version Client: Version: 1.13.1 API version: 1.26 Go version: go1.6.2 Git commit: 092cba3 Built: Thu Nov 2 20:40:26 2017 OS/Arch: linux/386 Server: Version: 1.13.1 API version: 1.26 (minimum version 1.12) Go version: go1.6.2 Git commit: 092cba3 Built: Thu Nov 2 20:40:26 2017 OS/Arch: linux/386 Experimental: false

…

On Wed, Jan 17, 2018 at 2:06 PM, Jonathan Claudius ***@***.*** > wrote: @scmaccal <https://github.com/scmaccal> the sudo/non-sudo behavior is simply a function of your current user not having docker privs. If you add this person to the docker group, I think that will solve the problem. As for the second issue, I'm wondering if it's a docker version issue, would you mind sharing the output of the following: docker-compose version docker version I'm assuming your version of docker doesn't support the AS capability, but I'd like to compare to be sure. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#121 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABYY7iQIHzGII5_Fl2-sqd0uE6vQiUA4ks5tLkS4gaJpZM4RYU4w> .

-- Scott

claudijd · 2018-01-17T19:34:30Z

@scmaccal I'm running docker-compose 1.18.0 and docker 17.12.0-ce. I'm wondering if upgrading both of those on your system will fulfill the FROM/AS behavior and start working.

SCMGuruLLC · 2018-01-17T20:43:55Z

I did some research and in order for me to install the latest version of Docker, I need to be running a 64-bit OS. I'll try this next. I suspect the problem will go away. We'll see.

…

On Wed, Jan 17, 2018 at 2:35 PM, Jonathan Claudius ***@***.*** > wrote: @scmaccal <https://github.com/scmaccal> I'm running docker-compose 1.18.0 and docker 17.12.0-ce. I'm wondering if upgrading both of those on your system will fulfill the FROM/AS behavior and start working. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#121 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABYY7vQGJn4fedorddSqAFPYCAnewGNBks5tLktHgaJpZM4RYU4w> .

-- Scott

SCMGuruLLC · 2018-01-18T00:23:14Z

I added the official repositories from Docker and installed the latest version of the docker and docker-compose packages. Everything works fine now on Ubuntu Server 16.04.3 LTS 64-bit. I recommend adding to the build requirements for Docker that only a 64-bit OS and the latest version of Docker will work. I think language about how the official repositories provided by Docker are best would be helpful too. Given that by default the Docker packages don't add a user to an elevated user group on Ubuntu and probably other Unix-like distributions, I recommend language about running sudo before the docker-compose command. Under Stand Up Infrastructure for number 1, I recommend adding: git clone https://github.com/mozilla/ssh_scan_api.git For number 2, add: git fetch git checkout postgres_sql_prototype I'm willing to maintain any building or other documentation for this or stable project. Just point me in the right direction. :) On Jan 17, 2018 3:43 PM, "Scott C. MacCallum" <maccallum.c.scott@gmail.com> wrote:

…

I did some research and in order for me to install the latest version of Docker, I need to be running a 64-bit OS. I'll try this next. I suspect the problem will go away. We'll see. On Wed, Jan 17, 2018 at 2:35 PM, Jonathan Claudius < ***@***.***> wrote: > @scmaccal <https://github.com/scmaccal> I'm running docker-compose > 1.18.0 and docker 17.12.0-ce. I'm wondering if upgrading both of those on > your system will fulfill the FROM/AS behavior and start working. > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > <#121 (comment)>, > or mute the thread > <https://github.com/notifications/unsubscribe-auth/ABYY7vQGJn4fedorddSqAFPYCAnewGNBks5tLktHgaJpZM4RYU4w> > . > -- Scott

claudijd · 2018-01-18T01:14:18Z

@scmaccal I have a doc for the typical docker infra setup, which is maintained on the GitHub wiki here (https://github.com/mozilla/ssh_scan_api/wiki/Deploying-ssh_scan_api-using-docker-compose). I'll probably add these additions there and then just ask people to follow that guide (since this PR will likely be deployed by EOW). Thanks for the feedback, that's really helpful to avoid future issues for people trying it out.

claudijd · 2018-01-18T01:19:36Z

@scmaccal the wiki instructions have been updated. For anyone wishing to test out this branch, they simply need to follow the instructions here and replace the git checkout master with git checkout postgres_sql_prototype and to let me know if they have any problems.

claudijd · 2018-01-18T01:53:25Z

@scmaccal you may have gotten a collaboration request on this repo. It was my attempt to list you as a named reviewer on this PR, but I guess that's not possible without giving you write access. I ended up canceling it because it didn't give me the desirer effect. I think your comments and help here have been excellent, thanks again for the help with this!

SCMGuruLLC · 2018-01-18T01:56:07Z

You're welcome. Happy to help. Scott

…

On Jan 17, 2018 8:53 PM, "Jonathan Claudius" ***@***.***> wrote: @scmaccal <https://github.com/scmaccal> you may have gotten a collaboration request on this repo. It was my attempt to list you as a named reviewer on this PR, but I guess that's not possible without giving you write access. I ended up canceling it because it didn't give me the desire effect. I think your comments and help here have been excellent, thanks again for the help with this! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#121 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABYY7rgPKS1CucFgmXmtF__Yyccd8OmOks5tLqQWgaJpZM4RYU4w> .

caggle · 2018-01-22T01:06:55Z

@claudijd Thanks. Yes I didn't realise the logic to handle socket timeout was actually in the proper client (ssh_scan).

Viewing the scan result for a scan attempted on a closed port (e.g. http://127.0.0.1:8000/api/v1/scan/results?uuid=0c435d0d-ddae-494d-a01d-b8c78bb5a01d) returns "status": "ERRORED", which is probably the expected result.

claudijd · 2018-01-22T14:54:11Z

Just an FYI: although this PR worked on the bench, in practice under a multi-threaded load, this PG port doesn't operate very well and is entirely too unstable to run in any predictable way. That said, I'm probably going to be rolling back these changes on the production server today and I'll be changing my strategy to leverage a better supported SQL abstraction, such as ActiveRecord. I've got the first commit of that effort started in #137 and I'll be shooting for a rough target of EOW to land the ActiveRecord port.

Jonathan Claudius added 3 commits January 9, 2018 14:28

First pass on postgres prototype

d3cde86

Remove mongo as a dependancy

cf317c3

remove collection accessor, copy pasta

f3cdd2c

claudijd mentioned this pull request Jan 9, 2018

Mock up a Postgres Prototype #120

Closed

Remove more copy pasta comments

fbac7a0

claudijd commented Jan 9, 2018

View reviewed changes

More progress on postgres prototype

c623b3c

claudijd changed the title ~~[WIP] First pass on postgres prototype~~ [WIP] postgres prototype Jan 10, 2018

Jonathan Claudius added 11 commits January 10, 2018 14:50

More progress on postgres update

9c993a0

Moar porting to postgress, just keep swimming

c899c2c

Remove setup stuff from schema.sql

ea37169

Moar updates, including time-based methods

b64a959

Remove stats object, it's no longer used

89192c1

Update API default config to match test expectations

bb2e6f0

Remove mongo as a DB type

697c0aa

Moar updated to postgres adapter

55d6988

Update database abstraction to reflect some minor changes to the inte…

42a3044

…rfaces

Remove stats require

8078eac

Update API expectations for queuing scans

2085e4c

Jonathan Claudius added 3 commits January 11, 2018 11:52

Remove mongo example from authenticator specs

8c0e90f

Add some docker DB bootstrap stuff

f8a5762

Finally able to properly task scans

71a04ed

Jonathan Claudius added 5 commits January 12, 2018 14:44

Now have a working docker setup, w00t

de2ea7c

Attempt to get unit-tests passing again on travis

58847c2

Make db init script executable

2fc871e

Fix copy pasta

5fe9a7e

Fix typo in DB init scripts

1386ec4

claudijd mentioned this pull request Jan 17, 2018

Test on new Ruby versions #129

Closed

claudijd changed the title ~~[ALPHA] Migrates SSH Observatory datastore from MongoDB to Postgres~~ [BETA] Migrates SSH Observatory datastore from MongoDB to Postgres Jan 17, 2018

Jonathan Claudius added 4 commits January 17, 2018 18:21

Fix postgres API expectations for unittests

223a18a

Minor tweaks to migration script

d694516

Attempt to upgrade postgres version to support JSONB

dc037b2

Remove unused config line

aa495e1

Strengthen unit-tests for API

fbb3a6b

Attempt to work around docker time shift bug

adc3e3a

Jonathan Claudius added 5 commits January 18, 2018 15:19

Remove commented code

990a03c

Last minute tweaks before deployment

bc54702

Revert secure_headers middleware upgrade

202e2ca

Fixing merge conflict from master

2ede435

Bump to 32

d584cae

claudijd changed the title ~~[BETA] Migrates SSH Observatory datastore from MongoDB to Postgres~~ Migrates SSH Observatory datastore from MongoDB to Postgres Jan 19, 2018

claudijd merged commit b3372f3 into master Jan 19, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Migrates SSH Observatory datastore from MongoDB to Postgres #121

Migrates SSH Observatory datastore from MongoDB to Postgres #121

claudijd commented Jan 9, 2018

claudijd Jan 9, 2018

claudijd Jan 9, 2018

claudijd commented Jan 11, 2018

claudijd commented Jan 12, 2018

caggle commented Jan 17, 2018

SCMGuruLLC commented Jan 17, 2018 •

edited

Loading

claudijd commented Jan 17, 2018 •

edited

Loading

claudijd commented Jan 17, 2018 •

edited

Loading

SCMGuruLLC commented Jan 17, 2018 via email

claudijd commented Jan 17, 2018

SCMGuruLLC commented Jan 17, 2018 via email

SCMGuruLLC commented Jan 18, 2018 via email •

edited

Loading

claudijd commented Jan 18, 2018

claudijd commented Jan 18, 2018 •

edited

Loading

claudijd commented Jan 18, 2018 •

edited

Loading

SCMGuruLLC commented Jan 18, 2018 via email

caggle commented Jan 22, 2018

claudijd commented Jan 22, 2018

Migrates SSH Observatory datastore from MongoDB to Postgres #121

Migrates SSH Observatory datastore from MongoDB to Postgres #121

Conversation

claudijd commented Jan 9, 2018

claudijd Jan 9, 2018

Choose a reason for hiding this comment

claudijd Jan 9, 2018

Choose a reason for hiding this comment

claudijd commented Jan 11, 2018

claudijd commented Jan 12, 2018

caggle commented Jan 17, 2018

SCMGuruLLC commented Jan 17, 2018 • edited Loading

claudijd commented Jan 17, 2018 • edited Loading

claudijd commented Jan 17, 2018 • edited Loading

SCMGuruLLC commented Jan 17, 2018 via email

claudijd commented Jan 17, 2018

SCMGuruLLC commented Jan 17, 2018 via email

SCMGuruLLC commented Jan 18, 2018 via email • edited Loading

claudijd commented Jan 18, 2018

claudijd commented Jan 18, 2018 • edited Loading

claudijd commented Jan 18, 2018 • edited Loading

SCMGuruLLC commented Jan 18, 2018 via email

caggle commented Jan 22, 2018

claudijd commented Jan 22, 2018

SCMGuruLLC commented Jan 17, 2018 •

edited

Loading

claudijd commented Jan 17, 2018 •

edited

Loading

claudijd commented Jan 17, 2018 •

edited

Loading

SCMGuruLLC commented Jan 18, 2018 via email •

edited

Loading

claudijd commented Jan 18, 2018 •

edited

Loading

claudijd commented Jan 18, 2018 •

edited

Loading