Skip to content

TalkLiberation/WLDragnet-Search

main
Switch branches/tags

Name already in use

A tag already exists with the provided branch name. Many Git commands accept both tag and branch names, so creating this branch may cause unexpected behavior. Are you sure you want to create this branch?
Code

Latest commit

 

Git stats

Files

Permalink
Failed to load latest commit information.
Type
Name
Latest commit message
Commit time
 
 
 
 
 
 
 
 
 
 
 
 
 
 

WLDragnet Logo

WLDragnet Search

https://wldragnet.com

This project is brought to you by the team building Panquake and is an example of the strong ethical design and high-quality standard we bring to the software development process. It is released as Free and Open-Source Software (FOSS) in the interest of the public good.

In October 2021, Talk Liberation published WLDragnet - an investigation that included 1500+ reports and a repository of 7,100+ files that prove US military contractors and weapons manufacturers are utilizing open-source intelligence to track and analyze the social media conversations of persons, hashtags, keywords and topics of political consequence, effectively creating target lists.

This profiling spans from 2018 up to the present day and its scope continues to grow. Nearly 200 reports and 500+ files were added to the WLDragnet repository in February 2022.

Featured in the target lists are whistleblowers, journalists, activists, and thousands of regular citizens. The files can be onerous to manually search through and difficult to understand, so Talk Liberation created a more high-level and stripped-down view: the WLDragnet Search.

Talk Liberation encourages readers to independently validate this work and to develop it further. The information here is provided for context as well as journalistic and technical exploration beyond the scope of our initial investigation. To support this important & innovative work, subscribe today: TalkLiberation.com

App Overview

This repository contains an application that complements investigative reporting into WLDragnet and makes it easy for Twitter users to find out if they have been targeted and in what context. They can then click through to the relevant sources for the full picture.

The WLDragnet Search is a Flask application that displays results generated from the WLDragnet files. This repository also includes utility scripts for scraping the data and a DDL file for the required database structure.

Privacy and Security

Users of the WLDragnet Search can look up their own Twitter accounts — or anyone else’s — without those searches being recorded by the application. Other privacy features include:

  • No cookies on the WLDragnet.com website
  • No web trackers
  • Search requests (inputted user handles) are not captured or stored
  • Reports generated via the tool are not captured or stored (in-memory data structure store)
  • No logging of browser, device or visitor information
  • Source links go to archived versions of the files — preventing the entities involved in the social media mass surveillance from tracking visitors
  • Hyperlinks to Twitter accounts in the custom reports lead to Nitter - a FOSS alternative Twitter front-end (hosted by @SethForPrivacy)

The code has been penetration tested by top security experts and audited by founder of Exodus Privacy, Esther Onfroy (also of Defensive Lab Agency).

Installation and Configuration

Dependencies

  • Python 3.9
  • Flask
  • Postgresql12.6+
  • Redis
  • pip
  • virtualenv
  • psycopg2
  • wkhtmltopdf
  • poppler

Docker Setup

docker-compose up

This will spawn three containers, wldragnet_redis, wldragnet_postgresql and wldragnet_flask. The application will then be available at http://127.0.0.1:5000/

In order to get data into the database, the folder documents/ will be mounted as a volume to the wldragnet_flask container. All PDF files in there can be scanned with

docker exec -it wldragnet_flask python3 /app/filescan/filescan.py

Manual Setup

The application needs a postgresql and a redis instance For testing and evaluation thepsycopg2-binary library from pip can be used instead of psycopg2 itself.

Preparation

  1. Download and install dependencies for psycopg2
  2. Download and install wkhtmltopdf
  3. Download and install pdftotext requirements
  4. Download and install Python
  5. Install pip
  6. pip install virtualenv

Installation

  1. Clone this repo
  2. cd into the cloned repo
  3. virtualenv venv
  4. source venv/bin/activate
  5. pip install -r dependencies.txt
  6. Connect to a postgres database and run the wldragnet-search-ddl.sql script to set up the database

Configuration

Create a .env file in the project root folder and edit accordingly. These weak credentials are for local testing and should be changed in other scenarios.

DOCUMENT_PATH=<path-to-graphs>
DOCUMENT_GLOB_PATTERN=**/*.html
SCAN_PROCESS_POOL_SIZE_MAX=4
DB_HOST=128.0.0.1
DB_PORT=5432
DB_NAME=wldragnet_search
DB_USER=wldragnet_search
DB_PW=<password>
DB_POOL_SIZE_MIN=4
DB_POOL_SIZE_MAX=4
JWT_SECRET_KEY=<arbitrary-secret>
JWT_TOKEN_LIFETIME=600
REDIS_HOST=localhost
REDIS_PORT=6379
REDIS_DB=0

Running the Application

Since this is a flask app, you can start the application from its vitualenv environment via

flask run

Licensing

This project is ethical Free and Open-Source Software (FOSS). Any and all original work contained in this repository that is authored by Talk Liberation is released under the GNU AGPL version 3 or any later version. See LICENSE for more information.

For JavaScript licensing information, see the LibreJS labels in weblabels.html

All of the data used to generate WLDragnet Search reports was gathered via technical analysis with freely-available and FOSS tools, and none of the published information was obtained via leaks, hacking, or data breaches.

The articles written on talkliberation.substack.com are © Copyright Talk Liberation CIC Limited and licensed under a Creative Commons Attribution-ShareAlike 4.0 International license. Contact Talk Liberation for further information or additional permissions.

Contact

For media inquiries, please contact media@talkliberation.com

About

WLDragnet Search

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published