Migration script

[mzfr]

This script can help people in migrating their data from old redis setup to new postgres model.

[coveralls]

Pull Request Test Coverage Report for Build 1425

• 10 of 47 (21.28%) changed or added relevant lines in 2 files are covered.
• 23 unchanged lines in 2 files lost coverage.
• Overall coverage decreased (-1.9%) to 75.992%

---

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
tanner/sessions/session_analyzer.py	2	3	66.67%
tanner/migration_script.py	8	44	18.18%

Files with Coverage Reduction	New Missed Lines	%
tanner/utils/asyncmock.py	2	85.71%
tanner/emulators/rfi.py	21	72.15%

Totals
Change from base Build 1392:	-1.9%
Covered Lines:	1570
Relevant Lines:	2066

---

💛 - Coveralls

[afeena]

you have r_client, not self.redis_client

[afeena]

while testing, I got

psycopg2.errors.NotNullViolation: null value in column "user_agent" violates not-null constraint
DETAIL:  Failing row contains (abd0fc36-d96d-4cba-8771-b5417bcc6cb1, 410c054a-3029-4e27-8172-c8679d75634a, 172.19.0.1, 50002, null, null, null, 0, null, 2019-10-12 02:52:00, 2019-10-12 02:52:00, 85598.04081632652, 0, 1, 0, 0, null).

You should validate the data before moving it into the postgres

[afeena]

please consider checking all non-null values

psycopg2.errors.NotNullViolation: null value in column "user_agent" violates not-null constraint
DETAIL:  Failing row contains (50e5ce97-16d4-41be-8e61-ddaeb99d0ab6, 77ce8271-cd9e-4cc1-9cd3-b5b93098a157, 192.168.16.1, 36872, null, null, null, 0, null, 2019-07-22 07:32:24, 2019-07-22 07:32:24, 107546.2564102564, 0, 1, 0, 0, null).

[afeena]

Don't forget to set values according to their types too :)

psycopg2.errors.InvalidTextRepresentation: invalid input syntax for type integer: "N/A"
LINE 1: ...58', 2.8084954716092154, 0.3560633938033859, 231, 'N/A', 'N/... ```

[afeena]

@mzfr the problem not only in integer, but in all the types with missing values

psycopg2.errors.InvalidTextRepresentation: invalid input syntax for type double precision: "N/A"
LINE 1: ...47:46', '2019-07-26 07:47:46', 89240.51063829787, 'N/A', 1, ...

[mzfr]

Is it because of the Float and Integer?

[afeena]

in raw data, there s no atbr and rps,
rps -->requests_in_second
atbr --> approx_time_between_requests

[afeena]

In our system, zip code is Integer, but I found out that it's not correct, e.g. in Japan, zip codes contain - and some US states have 0 at the beginning of the ZIP code, which is not a valid integer. So I'd recommend changing the type to string

[mzfr]

@afeena should I update the code as well as the schema for the postgres?

[afeena]

@mzfr yes, please go ahead :)

[afeena]

@mzfr I tested it and found some problems of migration, to make it easier, I will merge this PR and upload the changes as a separate commit :)