Skip to content
This repository has been archived by the owner on Feb 14, 2024. It is now read-only.

Create pe_source module and add cybersixgill class #117

Merged
merged 34 commits into from
Mar 8, 2022
Merged

Create pe_source module and add cybersixgill class #117

merged 34 commits into from
Mar 8, 2022

Conversation

aloftus23
Copy link
Contributor

@aloftus23 aloftus23 commented Jan 9, 2022 •
edited

🗣 Description

This PR introduces the pe_source as a moduel with the cybersixgill class.

pe_source module which will include cybersixgill, dnstwist, hibp, shodan and other data source scripts.

fixes #74
fixes #103

💭 Motivation and context

This will allow P&E to run our data collection scripts as a module.

🧪 Testing

Wrote new pytests and will make sure all lgtm and pre-commit checks are passed.

✅ Pre-approval checklist

  • This PR has an informative and human-readable title.
  • Changes are limited to a single goal - eschew scope creep!
  • All future TODOs are captured in issues, which are referenced
    in code comments.
  • All relevant type-of-change labels have been added.
  • I have read the CONTRIBUTING document.
  • These code changes follow cisagov code standards.
  • All relevant repo and/or project documentation has been updated
    to reflect the changes in this PR.
  • Tests have been added and/or modified to cover the changes in this PR.
  • All new and existing tests pass.

@aloftus23
Create pe_source module and add cybersixgill scripts
da00071 
- framework for pe_source module
- will include cybersixgill, dnstwist, hibp, shodan scripts and more
- this PR introduces the framework as well as the sixgill scripts
@aloftus23 aloftus23 added the improvement This issue or pull request will add or improve functionality, maintainability, or ease of use label Jan 9, 2022
@aloftus23 aloftus23 self-assigned this Jan 9, 2022
@lgtm-com
Copy link

lgtm-com bot commented Jan 9, 2022

This pull request introduces 2 alerts when merging da00071 into a76ccf4 - view on LGTM.com

new alerts:

  • 2 for Variable defined multiple times

@aloftus23
Add db_queries
a363597 
- add db_query.py which contains functions to insert data into the pe postgres db
@lgtm-com
Copy link

lgtm-com bot commented Jan 10, 2022

This pull request introduces 2 alerts when merging a363597 into a76ccf4 - view on LGTM.com

new alerts:

  • 2 for Variable defined multiple times

@aloftus23
Address lgtm errors, test errors and clean up logging
e3d5237 
- took out a duplicate concat() (lgtm)
- In pe_source/data/pe_db/config.py added a exception if the .ini can't be found
- address .replace() warning in pe_db/db_query.py
@aloftus23
Add empty .ini file
13447d3 
- empty .ini file
@coveralls
Copy link

coveralls commented Jan 10, 2022 •
edited

Pull Request Test Coverage Report for Build 1952507676

  • 0 of 0 changed or added relevant lines in 0 files are covered.
  • No unchanged relevant lines lost coverage.
  • Overall coverage remained the same at 17.348%
Totals Coverage Status
Change from base Build 1773811980: 0.0%
Covered Lines: 139
Relevant Lines: 766
💛 - Coveralls

@aloftus23
Clean up unused sixgill functions
2d8153a 
- deleted all sixgill functions in api.py that are unused.
- the full script can still be found in PE documents
@aloftus23
Update ReadMe
262dd5a 
- update readme to include the pe_source info
@aloftus23
fix readme typo
c958ee7
@aloftus23
Take out temporary org filter
ffce319 
- take out filter that was used for testing
@aloftus23 aloftus23 marked this pull request as ready for review January 14, 2022 13:36
@stewartl97 stewartl97 added this to PR in Sprint 9 (1/16 - 3/1) via automation Jan 20, 2022
@stewartl97 stewartl97 added the Tech Debt Issue or PR from Previous Sprints label Jan 20, 2022
@aloftus23
Merge branch 'dj-WIP-pages' into AL-cybersixgill
d18d728
@aloftus23
Update code based on dev suggestions in last PR
7f7199c 
- replace f-string logging with % operators
- fix test_pe_source
- make sure there are no spelling mistakes
@DJensen94 DJensen94 mentioned this pull request Feb 9, 2022
Closed
5 tasks
cduhn17
cduhn17 suggested changes Feb 24, 2022
View reviewed changes
Copy link
Collaborator

@cduhn17 cduhn17 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See the few items to consider.

src/pe_reports/data/config.py Outdated Show resolved Hide resolved
src/pe_reports/data/config.py Outdated Show resolved Hide resolved
src/pe_reports/data/config.py Outdated Show resolved Hide resolved
src/pe_source/data/pe_db/config.py Outdated Show resolved Hide resolved
src/pe_source/data/sixgill/config.py Outdated Show resolved Hide resolved
@cduhn17
Copy link
Collaborator

cduhn17 commented Feb 28, 2022

@dav3r ,
Do you have a time frame of when you might have some time to review this PR?
Thanks

@dav3r
Copy link
Member

dav3r commented Mar 1, 2022

@dav3r , Do you have a time frame of when you might have some time to review this PR? Thanks

@cduhn17 I will try to review it sometime this week. If possible, I'd prefer that this branch be up-to-date with the develop branch before I start reviewing it- can you please take care of that?

@aloftus23
Use centralized config file
a45d956 
- Have pe_source module use the config function already created in pe_reports
aloftus23 and others added 8 commits March 7, 2022 12:21
@aloftus23 @dav3r
Fix typos, redundancies and unnecessary lines in db_query.py
2b6e250 
Co-authored-by: dav3r <david.redmin@trio.dhs.gov>
@aloftus23
Add last_run update to get_data_source_uid function
e47d876
@aloftus23
Run lint on cybersixgill.py
74653a1
@aloftus23 @dav3r
Make test names more relevant to the module
31031c8 
Co-authored-by: dav3r <david.redmin@trio.dhs.gov>
@aloftus23 @dav3r
Fix remaining typos and small changes to style
c569955 
+= 1 instead of x = x + 1
f-strings

Co-authored-by: dav3r <david.redmin@trio.dhs.gov>
@aloftus23
Remove unnecessary __init__ files in data, pe_db, and sixgill folders
3eb78d8
@aloftus23
Clean up method to get path and logging in sixgill's config.py
cb34664 
Use importlib to fetch sixgill credentials in config.py and log path when database.ini file can't be found
@aloftus23 @dav3r
Use break in mentions function instead of explicitly "breaking" the c…
1dfdd6d 
…ondition

Co-authored-by: dav3r <david.redmin@trio.dhs.gov>
@aloftus23 @dav3r
Clean up count variable in mentions function
946c126 
Co-authored-by: dav3r <david.redmin@trio.dhs.gov>
@aloftus23 aloftus23 requested review from dav3r and removed request for cduhn17 March 8, 2022 14:43
dav3r
dav3r reviewed Mar 8, 2022
View reviewed changes
src/pe_source/cybersixgill.py Outdated Show resolved Hide resolved
dav3r
dav3r reviewed Mar 8, 2022
View reviewed changes
src/pe_source/cybersixgill.py Outdated Show resolved Hide resolved
@aloftus23
Simplify return statements
0a1df35 
- I prefer to keep the rest "as is" for the purpose of readability
dav3r
dav3r approved these changes Mar 8, 2022
View reviewed changes
Copy link
Member

@dav3r dav3r left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for considering all of my suggestions! 🚀

@aloftus23
Copy link
Contributor Author

Thanks for considering all of my suggestions! 🚀

Of course, thank you!

@cduhn17 cduhn17 merged commit 04281f2 into develop Mar 8, 2022
Sprint 9 (1/16 - 3/1) automation moved this from PR to Done Mar 8, 2022
@cduhn17 cduhn17 deleted the AL-cybersixgill branch March 8, 2022 17:08
aloftus23 pushed a commit that referenced this pull request Mar 30, 2022
@cduhn17
Merge pull request #117 from cisagov/AL-cybersixgill
f60b77a 
Create pe_source module and add cybersixgill class
aloftus23 pushed a commit that referenced this pull request Mar 30, 2022
@cduhn17
Merge pull request #117 from cisagov/AL-cybersixgill
82c7a61 
Create pe_source module and add cybersixgill class
aloftus23 pushed a commit that referenced this pull request Apr 13, 2022
@cduhn17
Merge pull request #117 from cisagov/AL-cybersixgill
81ada36 
Create pe_source module and add cybersixgill class
aloftus23 pushed a commit that referenced this pull request Apr 13, 2022
@cduhn17
Merge pull request #117 from cisagov/AL-cybersixgill
f1a12f5 
Create pe_source module and add cybersixgill class
cisagovbot pushed a commit that referenced this pull request Jan 7, 2023
@mcdonnnj
Merge pull request #117 from cisagov/improvement/update_yamllint_line…
cfa9ba5 
…-length_rules

Update the `line-length` configuration for `yamllint`
cisagovbot pushed a commit that referenced this pull request Jan 8, 2023
@mcdonnnj
Merge pull request #117 from cisagov/improvement/add_github_actions_a…
eb2605b 
…ttributions

Add ignore directives to the dependabot configuration
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@dav3r dav3r dav3r approved these changes

@cduhn17 cduhn17 cduhn17 approved these changes

@DJensen94 DJensen94 Awaiting requested review from DJensen94 DJensen94 is a code owner

@felddy felddy Awaiting requested review from felddy felddy is a code owner

@jsf9k jsf9k Awaiting requested review from jsf9k jsf9k is a code owner

@mcdonnnj mcdonnnj Awaiting requested review from mcdonnnj mcdonnnj is a code owner

@stewartl97 stewartl97 Awaiting requested review from stewartl97 stewartl97 is a code owner

Assignees

@aloftus23 aloftus23

Labels
improvement This issue or pull request will add or improve functionality, maintainability, or ease of use Tech Debt Issue or PR from Previous Sprints
Projects
No open projects
Sprint 9 (1/16 - 3/1)
Done
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Import CyberSixGill Classes from Crossfeed Repo Add pe-source module.
5 participants
@aloftus23 @coveralls @cduhn17 @dav3r @stewartl97