AMoC Hackathon

     _______________________________________
    < SamSepi0l's AMoC 2021 Hackathon Entry >
     ---------------------------------------
            \   ^__^
             \  (oo)\_______
                (__)\       )\/\
                    ||----w |
                    ||     ||

Sometime Github refuses to render jupyter notebooks; use this nbviewer page to see the notebooks in the browser with no live python kernel.

To run the dashboard, make sure you've run the data download targets from the Makefile, and make sure you're running a container with host networking because the Panel Server chooses a random port. (network in Makefile target)

Context:

The following is taken verbatim from the Hackathon description:

Bristol Cyber Security Group is hosting a cybercrime hackathon. Groups from select institutions are invited to create teams of 4 to tackle a three-day intensive hackathon.

The hackathon is aligned with our AMoC project (Advanced Modelling of Cyber Criminal Careers), which aims to tackle financially-motivated cybercrime on Darknet Markets by gaining a better understanding of cyber offenders, their motives, the interventions effective at disrupting their activities, the different roles within cybercriminal networks and how these evolve over time. Key outputs from the project will contribute to a software toolkit designed to assist law enforcement in their cybercrime investigations.

Teams will compete against each other on two data-driven cybercrime investigation tasks exploiting timestamped textual observations of underground cryptomarkets:

1. Longitudinal visualisation: Create an innovative visualisation representing the datasets over time, with particular attention to how significant events might affect the nature of posting on the forums.

2. Author re-identification: Train a competitive classifier to re-identify individuals' accounts across different forums, drawing on stylometric and temporal features.

Team Members:

• Lukas Gienapp: Affilitation

• Christopher Akiki: Affiliation 1 / Affilitation 2

Project Hierarchy

Local output of tree -s --charset X . :

Tree

.
|-- [       4096]  data
|   |-- [  575721195]  base.parquet
|   |-- [ 9994088576]  body_bertweet_embeddings.npy
|   |-- [ 6662725760]  body_embeddings.npy
|   |-- [ 6662725760]  category_embeddings.npy
|   |-- [        268]  download.sh
|   |-- [       4096]  hackathon_dataset
|   |   |-- [       4096]  fora
|   |   |   |-- [   80475006]  abraxas.csv
|   |   |   |-- [   59372422]  agora.csv
|   |   |   |-- [   13651267]  alphabay.csv
|   |   |   |-- [     532069]  andromeda.csv
|   |   |   |-- [   27654884]  blackbank.csv
|   |   |   |-- [   39761017]  bmr.csv
|   |   |   |-- [    1969455]  bungee54.csv
|   |   |   |-- [    3901035]  cannabisroad2.csv
|   |   |   |-- [    3164247]  cannabisroad3.csv
|   |   |   |-- [   10165330]  crackingarena.csv
|   |   |   |-- [   12338244]  crackingfire.csv
|   |   |   |-- [     215033]  darkbay.csv
|   |   |   |-- [     158679]  darknetheroes.csv
|   |   |   |-- [    2441620]  diabolus.csv
|   |   |   |-- [     167220]  dogeroad.csv
|   |   |   |-- [  267554251]  evolution.csv
|   |   |   |-- [      18226]  greyroad.csv
|   |   |   |-- [    2221113]  hackhound.csv
|   |   |   |-- [     117507]  havana.csv
|   |   |   |-- [     454402]  hydra.csv
|   |   |   |-- [    8364072]  kingdom.csv
|   |   |   |-- [     766236]  kiss.csv
|   |   |   |-- [      18774]  mrniceguy.csv
|   |   |   |-- [   42652560]  nucleus.csv
|   |   |   |-- [     884519]  outlawmarket.csv
|   |   |   |-- [    1935573]  panacea.csv
|   |   |   |-- [   30481136]  pandora.csv
|   |   |   |-- [    5037959]  pbf.csv
|   |   |   |-- [     398731]  revolver.csv
|   |   |   |-- [  444288998]  silkroad2.csv
|   |   |   |-- [  544526908]  silkroad.csv
|   |   |   |-- [   40346538]  thehub.csv
|   |   |   |-- [   22266519]  themajesticgarden.csv
|   |   |   |-- [     617373]  therealdeal.csv
|   |   |   |-- [      85842]  tom.csv
|   |   |   |-- [    1390316]  torbazaar.csv
|   |   |   |-- [     307867]  torescrow.csv
|   |   |   |-- [      17882]  tortuga.csv
|   |   |   |-- [    7229472]  utopia.csv
|   |   |   `-- [  833656446]  webkill.csv
|   |   |-- [     592443]  hackathon_tasks_overview.html
|   |   |-- [    2868111]  hackathon_tasks_overview.pdf
|   |   |-- [       4096]  measure_impact
|   |   |   `-- [     990714]  fora_registedusers.csv
|   |   `-- [       4096]  reidentification
|   |       |-- [     240999]  test_dataset.csv
|   |       |-- [     261243]  test_dataset.json
|   |       |-- [     938646]  train_negative_users.csv
|   |       |-- [    1017494]  train_negative_users.json
|   |       |-- [      25657]  train_positive_users.csv
|   |       `-- [      27783]  train_positive_users.json
|   |-- [       4096]  ngrams
|   |   |-- [   34761549]  bigrams.json
|   |   |-- [   18799932]  fourgrams.json
|   |   `-- [   17436096]  trigrams.json
|   |-- [ 6662725760]  subject_embeddings.npy
|   `-- [       4096]  tsne_XY
|       |-- [    8434055]  abraxasforums_perplexity_10_x_y.parquet
|       |-- [    2639180]  agoraforums_perplexity_10_x_y.parquet
|       |-- [    1021387]  blackbankmarketforums_perplexity_10_x_y.parquet
|       |-- [     105751]  bungee54forums_perplexity_10_x_y.parquet
|       |-- [     183134]  cannabisroad2_perplexity_10_x_y.parquet
|       |-- [     153972]  cannabisroad3_perplexity_10_x_y.parquet
|       |-- [      10490]  dogeroadforums_perplexity_10_x_y.parquet
|       |-- [   15847006]  evolutionforums_perplexity_10_x_y.parquet
|       |-- [       1924]  greyroadforums_perplexity_10_x_y.parquet
|       |-- [     132947]  hackhound_perplexity_10_x_y.parquet
|       |-- [      30442]  kissforums_perplexity_10_x_y.parquet
|       |-- [      22803]  outlawmarketforums_perplexity_10_x_y.parquet
|       |-- [      71196]  panaceaforums_perplexity_10_x_y.parquet
|       |-- [    1527708]  pandoraforums_perplexity_10_x_y.parquet
|       |-- [     192278]  projectblackflagforums_perplexity_10_x_y.parquet
|       |-- [      21831]  revolverforums_perplexity_10_x_y.parquet
|       |-- [    1828187]  thehubforums_perplexity_10_x_y.parquet
|       |-- [     814783]  themajesticgardenforums_perplexity_10_x_y.parquet
|       |-- [      34099]  therealdealforums_perplexity_10_x_y.parquet
|       |-- [       2264]  tortugaforums_perplexity_10_x_y.parquet
|       `-- [    4042036]  webkill_perplexity_10_x_y.parquet
|-- [       4096]  dockerfiles
|   |-- [       4096]  gpu
|   |   `-- [        755]  Dockerfile
|   `-- [       4096]  vanilla
|       `-- [       1368]  Dockerfile
|-- [       4096]  docs
|   |-- [        489]  commands.rst
|   |-- [       7878]  conf.py
|   |-- [        256]  getting-started.rst
|   |-- [        441]  index.rst
|   |-- [       5112]  make.bat
|   `-- [       5596]  Makefile
|-- [      11357]  LICENSE
|-- [       2231]  Makefile
|-- [       4096]  models
|-- [       4096]  notebooks
|   |-- [       2361]  2.0-lg-embeddings.ipynb
|   |-- [    3183370]  dashboard.ipynb
|   |-- [      17720]  document_embeddings_with_LMs.ipynb
|   |-- [      13984]  hackathon_tasks_overview.ipynb
|   |-- [    4509773]  kl_evolution.ipynb
|   |-- [       9480]  ngrams.ipynb
|   |-- [       6884]  nucleus_size_series.ipynb
|   |-- [      21050]  tsne.ipynb
|   |-- [    3118021]  umap.ipynb
|   `-- [       2260]  users.ipynb
|-- [       2758]  README.md
|-- [       4096]  references
|-- [       4096]  reports
|   `-- [       4096]  figures
`-- [       4096]  src
    |-- [       4096]  data
    |   `-- [        987]  generate-base-dataset.py
    |-- [          0]  __init__.py
    |-- [       4096]  __pycache__
    |   `-- [       1348]  utils.cpython-36.pyc
    `-- [       1283]  utils.py

Makefile targets examples (See the Makefile for more targets)

Regular Image:

foo@bar:/amoc-hackathon$ make vanilla-docker-build-image

foo@bar:/amoc-hackathon$ make vanilla-docker-run-jupyter

GPU Image:

foo@bar:/amoc-hackathon$ make gpu-docker-build-image

foo@bar:/amoc-hackathon$ make gpu-docker-run-jupyter

NVIDIA RAPIDS Image:

foo@bar:/amoc-hackathon$ make docker-run-rapids

Download the dataset as a consolidated Parquet dataframe:

foo@bar:/amoc-hackathon$ make download-base

Download the dasbhoard Parquet dataframe:

foo@bar:/amoc-hackathon$ make download-dashboard-dataset

Downloading the embeddings:

foo@bar:/amoc-hackathon$ make download-embeddings-body

Project based on the cookiecutter data science project template. #cookiecutterdatascience