Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

simple email generator #9

Merged
merged 14 commits into from
Jul 29, 2020
Merged

simple email generator #9

merged 14 commits into from
Jul 29, 2020

Conversation

Enhisir
Copy link
Contributor

@Enhisir Enhisir commented Jul 15, 2020

No description provided.

@Enhisir Enhisir added the enhancement New feature or request label Jul 15, 2020
@Enhisir Enhisir self-assigned this Jul 15, 2020
@HochuOlivie
Copy link
Contributor

Методы обычно пишутся с маленькой буквы, а классы с большой.

@Enhisir
Copy link
Contributor Author

Enhisir commented Jul 16, 2020

Методы обычно пишутся с маленькой буквы, а классы с большой.

fixed (5cb7414)

@manmolecular manmolecular added the big-pull-request Requires more time to check label Jul 17, 2020
@Enhisir Enhisir removed the enhancement New feature or request label Jul 19, 2020
Copy link
Member

@manmolecular manmolecular left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Сама обработка юзернейма крутая, и мутации прикольные, исходя из того, что я вижу в результатах!

Есть некоторые фундаментальные проблемы:

  • Во-первых, нам не нужно 673 домена. Нам достаточно взять топ-10 или топ-50, для начала: google, yandex, mail, yahoo, microsoft, etc., без каких-нибудь ChessFan.net и других специфичных
  • Максимальное число e-mail-ов не должно превышать 1.000, в идеале 500 адресов, чтобы мы могли это потом безболезненно скормить email_verifier модулю и узнать, какие из них существуют (и при этом сделать это относительно быстро, но это потом)
  • Реверс имени типа "john.doe.adams" -> "adams.doe.john" это круто, но надо оценить, насколько это применимо в реальной жизни? Как я вижу, такие перестановки могут очень сильно увеличить конечный пул результатов, поэтому, может их пока лучше "отключить" и посмотреть на количество с обычными мутациями типа -, _, ., etc.?
  • Модуль нужно привести к новому виду, как остальные скрипты
  • Модуль нужно перенести в папку convert/, т.к. он будет запускаться немного иначе и будет своего рода прослойкой для генерации результатов
  • Код из __init__.py нужно убрать в module.py в рамках приведения кода к новому виду

После этого можно вернуться к самому алгоритму обработки. А так, в общем, круто! Буду ждать изменений.

@Enhisir
Copy link
Contributor Author

Enhisir commented Jul 23, 2020

Сама обработка юзернейма крутая, и мутации прикольные, исходя из того, что я вижу в результатах!

Есть некоторые фундаментальные проблемы:

  • Во-первых, нам не нужно 673 домена. Нам достаточно взять топ-10 или топ-50, для начала: google, yandex, mail, yahoo, microsoft, etc., без каких-нибудь ChessFan.net и других специфичных
  • Максимальное число e-mail-ов не должно превышать 1.000, в идеале 500 адресов, чтобы мы могли это потом безболезненно скормить email_verifier модулю и узнать, какие из них существуют (и при этом сделать это относительно быстро, но это потом)
  • Реверс имени типа "john.doe.adams" -> "adams.doe.john" это круто, но надо оценить, насколько это применимо в реальной жизни? Как я вижу, такие перестановки могут очень сильно увеличить конечный пул результатов, поэтому, может их пока лучше "отключить" и посмотреть на количество с обычными мутациями типа -, _, ., etc.?
  • Модуль нужно привести к новому виду, как остальные скрипты
  • Модуль нужно перенести в папку convert/, т.к. он будет запускаться немного иначе и будет своего рода прослойкой для генерации результатов
  • Код из __init__.py нужно убрать в module.py в рамках приведения кода к новому виду

После этого можно вернуться к самому алгоритму обработки. А так, в общем, круто! Буду ждать изменений.

fixed(513b800)

@Enhisir Enhisir changed the title Made simple email generator, modified requirements.txt simple email generator Jul 23, 2020
Copy link
Member

@manmolecular manmolecular left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Попробуй запустить как модуль - не работает. Пути ломаные.

(venv) ➜  convert git:(email_generator) ✗ pwd
/osint-framework/src/scripts/convert
(venv) ➜  convert git:(email_generator) ✗ python3 -m email_generator test.this.name
{'message': "[Errno 2] No such file or directory: 'settings/settings.json'",
 'result': None,
 'status': 'error'}

Остальные замечания отметил. Буду ждать исправлений!

"""
:param domen_base: base of email domens written in translit as text file
"""
with open("settings/settings.json", "r", encoding="utf-8") as jfile:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Путь не относительный. Надо исходить от пути переменной __file__.

with open(Path(__file__).parent.joinpath("settings/settings.json"), mode="r") as settings_file:

"""
:param domen_base: base of email domens written in translit as text file
"""
with open("settings/settings.json", "r", encoding="utf-8") as jfile:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

jfile - непонятная переменная. Лучше тогда либо input_json, либо json_file, либо settings_json

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

наверное, стоит отказаться от файла json, так как настроек там всего 2, и их можно перенести в сам код

:param domen_base: base of email domens written in translit as text file
"""
with open("settings/settings.json", "r", encoding="utf-8") as jfile:
data = loads(jfile.read())
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

У JSON есть метод load, надо заменить loads на load, он предназначен для файлов

"""
with open("settings/settings.json", "r", encoding="utf-8") as jfile:
data = loads(jfile.read())
self.__domen_base = domen_base if domen_base is not None else data["domenBase"]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

self.__domen_base = domen_base or data.get("domenBase")

with open("settings/settings.json", "r", encoding="utf-8") as jfile:
data = loads(jfile.read())
self.__domen_base = domen_base if domen_base is not None else data["domenBase"]
self.__serv_syms = serv_syms if serv_syms is not None else data["serviceSymbols"]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

serv_syms or data.get("serviceSymbols")

with open("settings/settings.json", "r", encoding="utf-8") as jfile:
data = loads(jfile.read())
self.__domen_base = domen_base if domen_base is not None else data["domenBase"]
self.__serv_syms = serv_syms if serv_syms is not None else data["serviceSymbols"]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Эти разделители можно не задавать, они есть в стандартной библиотеке.

>>> from string import punctuation
>>> punctuation
'!"#$%&\'()*+,-./:;<=>?@[\\]^_`{|}~'

Но это только для разделения. Сборка все равно по ., _, -

И вообще, эти значения можно вынести на уровень модуля в класс "DefaultValues".

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

то есть, стоит использовать класс "DefaultValues" в качестве namespace?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

punctuation можно юзать напрямую

Comment on lines 24 to 27
for sym in self.__serv_syms:
if sym != '' and username.find(sym) != -1:
username = username.replace(sym, '_')
return username.split('_')
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

from string import punctuation
        for symbol in punctuation:
            username = username.replace(symbol, " ")
        return username.split()

login_base.extend([sym.join(parts) for sym in self.__serv_syms])
parts.reverse()
for login in login_base:
with open(self.__domen_base, "r") as domens:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Снимок экрана 2020-07-25 в 01 28 58

for login in login_base:
with open(self.__domen_base, "r") as domens:
for domen in domens:
yield f"{login}@{domen.strip()}"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Можно прочитать и сразу сделать splitlines, типа domens.read().splitlines()

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

И вообще не domen, а domain

… to a separate class; optimized code; made cosmetic changes
@Enhisir
Copy link
Contributor Author

Enhisir commented Jul 25, 2020

Попробуй запустить как модуль - не работает. Пути ломаные.

(venv) ➜  convert git:(email_generator) ✗ pwd
/osint-framework/src/scripts/convert
(venv) ➜  convert git:(email_generator) ✗ python3 -m email_generator test.this.name
{'message': "[Errno 2] No such file or directory: 'settings/settings.json'",
 'result': None,
 'status': 'error'}

Остальные замечания отметил. Буду ждать исправлений!

fixed(88c8025)

Copy link
Member

@manmolecular manmolecular left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Требуются небольшие фиксы.



class DefaultValues:
dividers = "!\"#$%&'()*+,-./:;<=>?@[\\]^_`{|}~"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

А зачем их задавать? Они же есть в string.

from string import punctuation as dividers

Можно их так напрямую и использовать, не добавляя в DefaultValues.

username = username.replace(sym, " ")
return username.split()

def generate(self, username: str) -> iter:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Результаты повторяются, если разделителей нет - надо исправить:

python3 -m convert.email_generator johndoe
{'message': 'Successfully finished! Got 96 logins, script lasted 0.00 seconds',
 'result': ['johndoe@yandex.ru', 'johndoe@yandex.com', 'johndoe@gmail.com',
            'johndoe@rocketmail.com', 'johndoe@yahoo.com',
            'johndoe@outlook.com', 'johndoe@example.com', 'johndoe@aol.com',
            'johndoe@love.com', 'johndoe@icloud.com', 'johndoe@inbox.com',
            'johndoe@mail.com', 'johndoe@yandex.ru', 'johndoe@yandex.com',
            'johndoe@gmail.com', 'johndoe@rocketmail.com', 'johndoe@yahoo.com',
            'johndoe@outlook.com', 'johndoe@example.com', 'johndoe@aol.com',
            'johndoe@love.com', 'johndoe@icloud.com', 'johndoe@inbox.com',
            'johndoe@mail.com', 'johndoe@yandex.ru', 'johndoe@yandex.com',
            'johndoe@gmail.com', 'johndoe@rocketmail.com', 'johndoe@yahoo.com',
            'johndoe@outlook.com', 'johndoe@example.com', 'johndoe@aol.com',
            'johndoe@love.com', 'johndoe@icloud.com', 'johndoe@inbox.com',
            'johndoe@mail.com', 'johndoe@yandex.ru', 'johndoe@yandex.com',
            'johndoe@gmail.com', 'johndoe@rocketmail.com', 'johndoe@yahoo.com',
            'johndoe@outlook.com', 'johndoe@example.com', 'johndoe@aol.com',
            'johndoe@love.com', 'johndoe@icloud.com', 'johndoe@inbox.com',
            'johndoe@mail.com', 'johndoe@yandex.ru', 'johndoe@yandex.com',
            'johndoe@gmail.com', 'johndoe@rocketmail.com', 'johndoe@yahoo.com',
            'johndoe@outlook.com', 'johndoe@example.com', 'johndoe@aol.com',
            'johndoe@love.com', 'johndoe@icloud.com', 'johndoe@inbox.com',
            'johndoe@mail.com', 'johndoe@yandex.ru', 'johndoe@yandex.com',
            'johndoe@gmail.com', 'johndoe@rocketmail.com', 'johndoe@yahoo.com',
            'johndoe@outlook.com', 'johndoe@example.com', 'johndoe@aol.com',
            'johndoe@love.com', 'johndoe@icloud.com', 'johndoe@inbox.com',
            'johndoe@mail.com', 'johndoe@yandex.ru', 'johndoe@yandex.com',
            'johndoe@gmail.com', 'johndoe@rocketmail.com', 'johndoe@yahoo.com',
            'johndoe@outlook.com', 'johndoe@example.com', 'johndoe@aol.com',
            'johndoe@love.com', 'johndoe@icloud.com', 'johndoe@inbox.com',
            'johndoe@mail.com', 'johndoe@yandex.ru', 'johndoe@yandex.com',
            'johndoe@gmail.com', 'johndoe@rocketmail.com', 'johndoe@yahoo.com',
            'johndoe@outlook.com', 'johndoe@example.com', 'johndoe@aol.com',
            'johndoe@love.com', 'johndoe@icloud.com', 'johndoe@inbox.com',
            'johndoe@mail.com'],
 'status': 'success'}

@@ -0,0 +1,12 @@
yandex.ru
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

добавить:

ya.ru
inbox.ru
list.ru
bk.ru
mail.ru

@Enhisir
Copy link
Contributor Author

Enhisir commented Jul 27, 2020

Требуются небольшие фиксы.

fixed(28c75db)

Copy link
Member

@manmolecular manmolecular left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, но нужно написать тесты. Создаем issue на эту задачу.

@manmolecular manmolecular merged commit 65d8c70 into develop Jul 29, 2020
@manmolecular manmolecular deleted the email_generator branch July 29, 2020 16:23
manmolecular added a commit that referenced this pull request Aug 14, 2020
* WIP: Add Initial Version (alpha-beta-gamma) (#1)

* Add initial WIP project version

* Update src code

* Update documentation

* Add requirements

* add script name_check (#4)

* Add initial WIP project version

* add check_nickname.py,
check_nickname_sync and check_nickname_async in it
add social_networks.txt

* add check_nickname.py,
check_nickname_sync and check_nickname_async in it
add social_networks.txt

* new main.py

* new base: develop
add many files

* fix inheritance in Runner

* fix check_nickname
fix social_networks.txt

* Little fixes

* Return default main value

Co-authored-by: manmolecular <regwebghost@yandex.ru>

* Add script: region check (#5)

* Added number to Possiblekeys in osint.py. Added number as kwarg in main.py. Script defines the region where the phone number is registrated

* Changed number->phone

* Little fixes, update requirements

Co-authored-by: manmolecular <regwebghost@yandex.ru>

* add script (email_verifier) (#7)

* add script (email_verifier)

* add library

* change req and so on

* used black

* Little fixes

* Main and .gitignore fixes

Co-authored-by: Minerm <minerm@mail.ru>
Co-authored-by: manmolecular <regwebghost@yandex.ru>

* Add script to get allowed http methods. (#10)

* Add script to get allowed http methods.

* 'Handle' more probable exceptions

* Changed name of script directory

* Remove wildcard import

* Add random method check.
Alter script response.

* Little fixes

* Fix methods variable

* Fix typo in requirements

Co-authored-by: manmolecular <regwebghost@yandex.ru>

* Add script to calculate hash of favicon.ico for Shodan (#13)

* Add script to calculate hash of favicon.ico

* Remove '\n'

* Change variable name

* Core fixes. Provide global variables. Update runner. Upgrade module-package system. (#14)

* Clean init all the modules with the __init__.py file

* Remove unused gitkeeps

* Add base inits for script packages

* Update main file

* Set __file__ variable at the module level

* Add inits for the modules

* Modify all the modules

* Delete one more .gitkeep

* Add WIP test for user_greeting

* Update test module for user_greeting

* Fix module runner a bit

* Add relative imports to the modules

* Add run tests script

* Update README.md

* Fix module-package runner

* Update requirements

* Format everything with Black

* Handle module and execution errors with runner

* Awesome cookie checker (#6)

* made cookie checker task

* added requirements.txt

* refactored the code & adapted to the new project structure

* small fix that processes exception

* Little fixes

Co-authored-by: manmolecular <regwebghost@yandex.ru>

* Added a function for retrieving location and provider info by IP address. (#12)

* Added IP address as one of the possible keys.

* Init commit.

* Init commit.

* Added dosctring.

* Removed some emptry lines.

* get_ip_info was moved into Runner class.

* The IP key shouldn't be here.

* Reworked exception handling mechanism.

* Codestyle fix.

* Removed some test code.

* Moved script to proper dir.

* Fixed response returned in case of error.

* Fixed exception handling.

* Request now uses the validated IP.

* Found a better variable name for validated IP.

* Removed unnecessary variables.

* Init commit.

* Runner class was moved into this module.

* Set default IP to Google DNS Resolver address (to be more stable)

* Add default values; set default ip to 8.8.8.8; set default city to Ashburn

* Apply Black linter on the main module file

Co-authored-by: manmolecular <regwebghost@yandex.ru>

* Add simple server mocking test example (#20)

* Add simple web server mocking example

* Add some types

* Add tests for favicon hash (#22)

* Add tests for favicon hash

* Apply Black linting

* Suppress server output logging

Co-authored-by: manmolecular <regwebghost@yandex.ru>

* Test allowed_methods. Modify allowed_methods. (#21)

* * Make allowed_methods also return methods on which server did not response.

* Add tests for allowed_methods module.

* Fix output

* Change docstring

* Linting with Black

Co-authored-by: manmolecular <regwebghost@yandex.ru>

* Fix tests timeouts (#25)

* Add tests defaults (#26)

* Boost up/improvement: Add multiprocessing CaseManager (processes, threads, parallel case execution) (#27)

* Add multiprocessing CaseManager

* Add more info about case to the manager

* Update logging format

* Set optimal threads and processes quantity

* simple email generator (#9)

* Made simple email generator, modified requirements.txt

* deleted __main__ rubbish, modified success message

* added a check on username availability

* Update __main__.py

* Update __main__.py

* modified names of methods and atributes

* Update requirements.txt

* reduced number of domen names, possible emails; reduced reversing and optimized code; moved project to convert/ dir; led project to a new view

* fixed bug with command prompt; got rid of json; moved service symbols to a separate class; optimized code; made cosmetic changes

* some fixes

* some fixes

* Add fixes

Co-authored-by: manmolecular <regwebghost@yandex.ru>

* Added phone number generator and normaliser script file (#11)

* Added phone number generator and normaliser script file

* Added feature directory and empty files

* Added module.py

* Added Runner class to main.py

* Cleaned up the comments and reformatted code

* Added library to the main requirements.txt

* Changed module.py

* Changed module.py

* Deleted old script file

* Fixed requirements bug

* Add phone_num_generator module

* Corrected double brackets and dot format mistakes

* Something went wrong last time, so no2

* Added unittests and local number format

* Fix main module runner

* Remove code duplicate

Co-authored-by: manmolecular <regwebghost@yandex.ru>

* Get title (#23)

* add main code

* add __init__.py,__main__.py,module.py,test_module.py

* fix module.py
fix test_module.py

* add test_module.py

* add requirements.txt

* fix tests,module.py
add new lines in __main__.py and __init__.py

* Remove old format string

* Delete test_module from check_nickname

* Module fixes

Co-authored-by: manmolecular <regwebghost@yandex.ru>

* Suppress Requests keep-alive socket warnings (#33)

* Iknowwhatyoudownload (#24)

* added torrent module

* underscores bugfix

* added docstrings

* updated requirements

* changed directory

* Added required changes

* Fix base runners

* Remove API key, fix main runner

* Linting with Black

Co-authored-by: omarkelov <36790281+omarkelov@users.noreply.github.com>
Co-authored-by: Anton Nikolaev <regwebghost@yandex.ru>

* add test_module.py in email_verifier (#29)

* add test_module.py

* Add functions (test_pass_false / true_arguments)

* Fix comparison

* Fix return types

Co-authored-by: Minerm <minerm@mail.ru>
Co-authored-by: manmolecular <regwebghost@yandex.ru>

* Fix validator keys (#35)

* Develop: fixes, improvements, etc. (#36)

* Suppress insecure request warning messages

* Update requirements.txt

* Add base scenario, add yaml support

* Update structure; include results saving

* Improvements - skip not applicable scripts, fix workers system (#39)

* Fix executors

* Optimize imports

* Add results to gitignore

* Add 'required' field to define required args

* Ignore results from 'results' directory

* Add 'required' field to base classes

* Skip not applicable scripts

* Fix quantity of processes

* Set the execution timeout to 5 mins per case

* Add phone case to the examples

* JSON Russian language and encoding support

* Add core default values

* Remove hardcoded quantity of workers

* Increase quantity of cases up to 10

* Fix indentation in runner

* Add seed (initial) data to scan for

* Email generator fix and tests (#38)

* added tests

* fixed incorrect behavior

* extension of tests, cosmetic changes

* Add google_search module (#37)

* Add module scraping search results.

* Add tests to google_search script

* Add tests and do minor refactoring

* Add input type test

* Apply black linting

* Add request delay when needed

* Remove redundant import

* Fixes (#40)

* Add 'required' field for the google search module

* Add beatifulsoup to requirements

* Check nickname (#28)

* add main code

* add __init__.py,__main__.py,module.py,test_module.py

* fix module.py
fix test_module.py

* add test_module.py

* add requirements.txt

* fix tests,module.py
add new lines in __main__.py and __init__.py

* add test_module.py

* delete get_title

* add some tests to test_module.py
fix social_networks.txt

* fix styles
modify social_networks.txt

* fix module.py
fix test_module.py

* Cosmetic fixes

Co-authored-by: manmolecular <regwebghost@yandex.ru>

* Update README.md

* Add Tornado-based web-server. Support REST API methods. Implement task manager.  (#41)

* Modify environment files

* Move saver to another module

* Add SQLite database handling

* Add Tornado server

* Add main server runner

* Move saver to another module

* Add multiprocessing

* Add docker-compose and postgres

* Update database handling

* Update environment files

* Remove unused imports from main.py

* Update database handlers

* Update task manager; handle task status

* Fix task spawner

* Little cosmetic fixes, format files

* Format with Black

* Add environment variables

* Add limit parameter to the task list handler

* Don't initialize task spawner, it's not required

* Remove single case wrapper from the main.py

* Add Docker support for server

* Fix makefile: spaces to tabs

* Format with Black

* Cosmetic: add new line to dockerignore

* Add healthcheckers

* Add CLI interface

* Add color logging

* Update gitignore

* Add parents for scripts in logging

* Fix makefile

* Add screenshots

* Update README.md

* Update README.md

Co-authored-by: Iandmee <48530455+Iandmee@users.noreply.github.com>
Co-authored-by: sph-668 <68100447+sph-668@users.noreply.github.com>
Co-authored-by: Neo <55306931+Shas08@users.noreply.github.com>
Co-authored-by: Minerm <minerm@mail.ru>
Co-authored-by: Nikita Kryukov <cravtos92@gmail.com>
Co-authored-by: HochuOlivie <49933393+HochuOlivie@users.noreply.github.com>
Co-authored-by: Timur <36272321+SN4KEBYTE@users.noreply.github.com>
Co-authored-by: Matvey Sergeev <47077074+Enhisir@users.noreply.github.com>
Co-authored-by: marinepalyan <50265835+marinepalyan@users.noreply.github.com>
Co-authored-by: katerina <42969775+edubinskaya18214@users.noreply.github.com>
Co-authored-by: omarkelov <36790281+omarkelov@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
big-pull-request Requires more time to check
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants