Add custom providers to Faker

[akarpf]

Problem Description

I created a custom Faker provider which I would like add to the Faker instance called by SDV.

Expected behavior

The faker methods defined in the custom provider are callable via "anonymize_fields".

Additional context

I asked the question on Slack and csala suggested me to add my provider to Faker globally right before going into the SDV model. I tried to do so:

from faker import Faker
from faker.providers import BaseProvider
fake = Faker()
fake.add_provider(CustomFaker)

I however got the following error message when I tried to fit the model:

ValueError: Category "faker_method" couldn't be found on faker

(If such a workaround is possible, I would be very grateful, if somebody could indicate me how!)

[csala]

Hi @akarpf thanks for opening this issue.

It looks like this could be achieved by writing your faker provider in a module and registering it globally.

For example, I can create a provider inside a my_providers folder as follows:

$ tree my_providers/
my_providers/
├── dummy.py
└── __init__.py

0 directories, 2 files
$ cat my_providers/dummy.py 
import random

from faker.providers import BaseProvider


# This has to be called exactly Provider, otherwise Faker does not work.
class Provider(BaseProvider):
    def dummy(self):
        return random.choice(['foo', 'bar', 'baz'])

Such that I should be able to import and run it from python:

In [1]: from my_providers.dummy import Provider
   ...: from faker import Faker
   ...: faker = Faker()
   ...: faker.add_provider(Provider)
   ...: faker.dummy()
Out[1]: 'foo'

Then, when I want to use it within SDV I just need to register the provider globally by appending its full module path to the faker.config.PROVIDERS list:

from faker.config import PROVIDERS

PROVIDERS.append('my_providers.dummy')

And afterwards use SDV normally just declaring dummy as my anonymization category when configuring anonymous fields.

[csala]

Let's also keep this issue open to add support for custom providers directly within SDV without having to follow the workaround explained above.

I think that it would be interesting to add the option to pass provider functions or full provider paths as the anonymization settings, so SDV would take care of adding the provider to the Faker instance on the fly without having to configure this outside of SDV beforehand.

[akarpf]

Hi @csala! Thank you very much! Your solution works perfectly and is also very convenient and clear. I can define multiple providers as methods inside that class and then call them from within SDV. In my opinion, it doesn't need any further integration. It would just be great if you could add this example to the documentation.

While exploring your solution I also found out how I can change the locale in a similar fashion:

import faker
#change locale
faker.proxy.DEFAULT_LOCALE = 'de_AT'
#add custom provider
faker.config.PROVIDERS.append('custom_provider.random_number')

Although it would be great, if I couldn't only change the DEFAULT_LOCALE but add a list of potential locales as it's possible with Faker(['en_US','de_AT']). Do you have an idea how I could achieve that?

[abmorte]

Hi @csala, @akarpf the code above work correctly for me as 'pt_BR' locale. Thanks!

[xamm]

Since I also stumbled upon the problem of needing multiple locales for an anonymized field, I would like to make a suggestion for the new feature and could also contribute a PR to it.

Suggestion:

Would it be possible to introduce a new metadata field like pii_locales in addition to pii_category to allow multiple locales when anonymizing columns?

In the pii_locales metadata description, users could specify an OrderedDict or a List of locales that should be used and passed to Faker() when it is created.

This should then result in faked values being generated from the specified pii_category depending on the locales specified in pii_locales.

Since the Faker() object is only created in the _get_faker function, it seems that only there the locales from pii_locales need to be passed.

The _get_faker function is called in two places and the pii_category is passed to it.
To make it backward compatible the locales could be added to the _get_faker function as optional parameter so that current usages of SDV won't be broken by the new feature.

[xamm]

I created the PR #609 to show what I meant in my previous comment.