This repository was archived by the owner on Feb 2, 2024. It is now read-only.
Refactor df.drop to improve compile times #945
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
kozlov-alexey
changed the title
kozlov-alexey
commented
Nov 11, 2020
kozlov-alexey
commented
Nov 11, 2020
kozlov-alexey
commented
Nov 11, 2020
kozlov-alexey
changed the title
commit 45e67860d0fc7c2339250764944253d45398e88f
Author: Kozlov, Alexey <alexey.kozlov@intel.com>
Date: Sat Nov 21 00:39:53 2020 +0300
Reverting back to use of list.initial_value, updating tests
commit 5b15f127f48e836b266c7817844978e1c28e2dbb
Author: Kozlov, Alexey <alexey.kozlov@intel.com>
Date: Tue Nov 17 01:47:49 2020 +0300
Trying LiteralList instead (works with typeinfer fix in numba)
commit b54d365e6ae4e735e9a738b9f41fcc28cad81472
Author: Kozlov, Alexey <alexey.kozlov@intel.com>
Date: Mon Nov 16 20:00:51 2020 +0300
Non-working case for changed list of dropped columns
commit 31768d874a645824633c2adfbe8a193870486835
Author: Kozlov, Alexey <alexey.kozlov@intel.com>
Date: Wed Nov 11 21:35:26 2020 +0300
Make df.drop aligned with pandas and take columns as list
kozlov-alexey
commented
Nov 23, 2020
AlexanderKalistratov
approved these changes
Dec 7, 2020
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation: old implementation of df.drop() produced LLVM IR of large size on DFs
with hundreds of columns, since it extracts all the columns from original DF and then packs
them into a internal structure (a tuple of lists of arrays) again. The new implementation
will make copy of internal df structure and just pop dropped columns from selected lists, which
heavily reduces IR size and compilation time.
Some numbers: