feature: private tlds can be used at call-time

[brycedrennan]

This is a second attempt at doing what was done in #144.

Addresses #66

• Adds include_psl_private_domains to the __call__ method. This is now something you can choose on a per-call basis. The object level argument now is only a default value for each call.
• The entire dataset from publicsuffix.org is saved to cache
• Ensured no weird cache issues happen when using with different suffix_list_urls by using different filenames per suffix_list_urls
• Use filelock to support multiprocessing and multithreading use cases
• Updates the bundled snapshot to be the raw publicsuffix data. Need to look at performance impact of this.
• various other cleanups
• Breaking change cache_file => cache_dir

[brycedrennan]

@john-kurkowski for your consideration. This replaces #144. I think it's cleaner than the first attempt.
@hangtwenty might also want to take a look

Hopefully we can merge this faster than the last PR 😄

[john-kurkowski]

Let's! 👍 😅 Will look later today.

[john-kurkowski]

Makes sense! Clean! Can merge in the next couple days.

[brycedrennan]

Nice catches. Addressed your feedback. Left as a separate commit but will squash it once you've looked at it

[brycedrennan]

Oh one thing I should point out is I deleted that snapshot diff functionality. Admittedly was just feeling a little lazy at figuring out how to implement that again. Can figure it out if it's important though.

[john-kurkowski]

Glad you pointed that out. I think it's fine without. I'll note in the CHANGELOG.

[john-kurkowski]

Thank you very much for this much requested feature, and seeing it through years!

[brycedrennan]

Thanks! I hope i didn’t sneak that multiprocessing test and related fix by you ( the last two commits)

[john-kurkowski]

Tests and fixes always welcome. I see no sneaking. 🙏

[evrial]

Fantastic changes, great job

In [1]: from tldextract import TLDExtract                                                                                                                                         

In [2]: TLDExtract.tlds                                                                                                                                                           
Out[2]: <property at 0x7faa2da45c50>

In [3]: t = TLDExtract()                                                                                                                                                          

In [4]: t.tlds                                                                                                                                                                    
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-4-2bd415795d39> in <module>
----> 1 t.tlds

~/anaconda3/envs/ocean/lib/python3.7/site-packages/tldextract/tldextract.py in tlds(self)
    243     @property
    244     def tlds(self):
--> 245         return self._get_tld_extractor().tlds
    246 
    247     def _get_tld_extractor(self):

AttributeError: '_PublicSuffixListTLDExtractor' object has no attribute 'tlds'

[brycedrennan]

🤨 hmm I suspect sarcasm

Yes it does seem we missed that. Given that we now track public and private tlds we probably won't be able to support this ambiguous property.

Perhaps a public_tlds and a private_tlds?

[evrial]

Yes, thats sensible names.

[evrial]

Are these results correct?

        (
            'runners.blogspot.com',
            {'subdomain': '', 'domain': 'runners', 'suffix': 'blogspot.com'},
        ),
        (
            'http://www.ocean.io',
            {'subdomain': 'www', 'domain': 'ocean', 'suffix': 'io'},
        ),
        (
            'blogspot.com',
            {'subdomain': '', 'domain': 'blogspot', 'suffix': 'com'},
        ),

Also this:

In [6]: tldextract.extract('runners.blogspot.com')                                                                                                                                
Out[6]: ExtractResult(subdomain='runners', domain='blogspot', suffix='com')

In [7]: tldextract.extract('runners.blogspot.com', include_psl_private_domains=True)                                                                                              
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-7-0e1c98efa124> in <module>
----> 1 tldextract.extract('runners.blogspot.com', include_psl_private_domains=True)

TypeError: extract() got an unexpected keyword argument 'include_psl_private_domains'

[brycedrennan]

I can't know whether those results are correct without knowing what calls produced them.

The second issue you pointed out is another oversight.

I've made a PR to address these issues here: #210

[evrial]

We were using wrapped patched version of tldextract to properly handle private domains and invoke tldextract with private parameter if the domain is private and vice versa. Above are our tests

    def is_private_domain(self, result):
        return (
            result.suffix in self.psl_private_tld_list and
            (
                not result.domain or result.domain == 'www'
            )
        )