Skip to content

Latest commit

 

History

History
201 lines (128 loc) · 3.9 KB

README.rst

File metadata and controls

201 lines (128 loc) · 3.9 KB
Raw

datanonymizer

Anonymizer tool for datasets such CSV files.

To generate fake data, you can choose between two excelent generators:

Install

Using pip:

pip install datanonymizer

Using mimesis instead of the default Faker:

pip install datanonymizer[mimesis]

Or from source:

git clone https://github.com/fgmacedo/datanonymizer
cd datanonymizer
python setup.py install

Usage

Pass your data through stdin and get it back anonymized on stdout.

Note

In this case, the output will be equal to the input as no conversions were applied.

cat input_file.csv | datanonymizer >output_file.csv

Using a config file to declare conversions and generators for the required fields:

cat input_file.csv | datanonymizer --config ./dataset_anon_config.yml >output_file.csv

Please see examples folder for a small demo:

cat examples/small.csv | python -m datanonymizer -i --config examples/small_faker.yml --seed my_seed >examples/small_anonymized_using_faker.csv

Optional arguments:

-h, --help            show this help message and exit
-l LANGUAGE, --language LANGUAGE
                      Language used by the Generator
-di DELIMITER_INPUT, --delimiter_input DELIMITER_INPUT
                      CSV delimiter
-do DELIMITER_OUTPUT, --delimiter_output DELIMITER_OUTPUT
                      CSV delimiter
-i, --ignore_errors   Continue on errors
--head HEAD           Outputs only the first <HEAD> lines
-g {faker,mimesis}, --generator {faker,mimesis}
                      Generator library to be used for fake data
--seed SEED           Seed for the pseudo random generator providers
--config CONFIG       Configuration file

Config file

You'l need a configuration file to setup transformations for each dataset.

This file is a simple yaml where you can configure fields.

Field names should match the column name declared into the CSV input file.

---
fields:
  Task ID:
    omit: true
  Location:
    conversions:
      - fn: coords_to_h3
        kwargs:
          resolution: 8
  Client Address:
    conversions:
      - fn: has_value
    rename: has_address
  Company Name:
    generator:
      provider: business.company
    rename: company
  Invoice ID:
    generator:
      provider: person.identifier
      kwargs:
        mask: "#######"
    rename: invoice

Generators

The generatos clause depends of the library you choose to provide fake data.

You can use any generator available at the generic API from Faker or mimesis.

For example, if you wanna mimic data with company names:

  • Faker

    ---
fields:
  Company Name:
    generator:
      provider: company

  • Mimesis

    ---
fields:
  Company Name:
    generator:
      provider: business.company

But you can replace the real names by names of fruits (using Mimesis) or any other provider:

---
fields:
  Company Name:
    generator:
      provider: food.fruit

Or generate random integers to replace real IDs:

  • Faker

    ---
fields:
  ID:
    generator:
      provider: pyint
      kwargs:
        min_value: 1
        max_value: 15_000_000

  • Mimesis

    ---
fields:
  ID:
    generator:
      provider: person.identifier
      kwargs:
        mask: "#######"

Conversions

You can apply any pre-configured conversion functions available.

  • coords_to_h3
  • has_value