Tabular: Enhance drop_duplicates, enable by default

[Innixma]

Issue #, if available:

Description of changes:

• Previously due to a bug in pandas (or otherwise very strange logic), duplicate category features with NaNs would cause a crash in DropDuplicatesFeatureGenerator because Series.replace({np.nan: foobar}) would not actually replace NaN, unlike if the dtype was numerical or object type.
  • Note: This bug did not cause crashes for standard users as DropDuplicatesFeatureGenerator was previously not enabled except for TextSpecial features which never contain NaN and are numeric
• This PR fixes the above issue by first converting category features to object when checking for duplicates.
• Also adds comprehensive unit testing for all of the edge-case scenario's involving NaN and mixed dtypes.
• Enabled post_drop_duplicates=True by default in PipelineFeatureGenerator, which means that TabularPredictor will now automatically filter duplicate columns (previously it did not do this). This will help significantly in cases where the user passes data with many duplicate features, and also helps to filter out redundant ngrams significantly when text is present. (You can see the ngram filtering in action with the changed AutoMLPipelineFeatureGenerator unit test outputs)
• Reduced the sample sizes used for duplicate detection to avoid overly long fit times for the generator. The chosen value (2000) should be high enough to still correctly de-dupe features in virtually all practical situations.

TODO:

• Benchmark on AMLB to ensure stability and performance

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

[Innixma]

@gradientsky This PR causes EDA unit tests to fail, probably because additional features are being pruned now in the feature generator.

[gradientsky]

@gradientsky This PR causes EDA unit tests to fail, probably because additional features are being pruned now in the feature generator.

Added fixes + merged main

[github-actions]

Job PR-3010-98ade7e is done.
Docs are uploaded to http://autogluon-staging.s3-website-us-west-2.amazonaws.com/PR-3010/98ade7e/index.html

[gradientsky]

LGTM