Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Optimization for text column profile ksneab #791

Merged
merged 3 commits into from
Apr 24, 2023
Merged

Optimization for text column profile ksneab #791

merged 3 commits into from
Apr 24, 2023

Conversation

ksneab7
Copy link
Contributor

@ksneab7 ksneab7 commented Apr 24, 2023 •
edited

The tests ran to generate these numbers can be found in the dataprofiler/tests/space_time_analysis/structured_space_time_analysis.py script

In repo dataset:

  • 0
    • Profile
      • Pre change
        • Total space allocated: N/A
        • Line specific space allocated (147): N/A
        • Profile runtime: 0.022213220596313477 s
      • Post change
        • Total space allocated: N/A
        • Line specific space allocated (147): N/A
        • Profile runtime: 0.020266056060791016 s
    • Merge
      • Pre change
        • Total space allocated: N/A
        • Line specific space allocated (147): N/A
        • Merge runtime: 9.059906005859375e-06 s
      • Post change
        • Total space allocated: N/A
        • Line specific space allocated (147): N/A
        • Merge runtime: 8.344650268554688e-06 s
  • 100
    • Profile
      • Pre change
        • Total space allocated: 960.1 KiB
        • Line specific space allocated (147): N/A
        • Profile runtime: 0.22062182426452637 s
      • Post change
        • Total space allocated: 960.1 KiB
        • Line specific space allocated (147): N/A
        • Profile runtime: 0.2196950912475586 s
    • Merge
      • Pre change
        • Total space allocated: 542 KiB
        • Line specific space allocated (147): N/A
        • Merge runtime: 0.02617621421813965 s
      • Post change
        • Total space allocated: 542 KiB
        • Line specific space allocated (147): N/A
        • Merge runtime: 0.0270388126373291 s
  • 1000
    • Profile
      • Pre change
        • Total space allocated: 680.4 KiB
        • Line specific space allocated (147): 193.3 KiB
        • Profile runtime: 0.2128901481628418 s
      • Post change
        • Total space allocated: 583 KiB
        • Line specific space allocated (147): N/A
        • Profile runtime: 0.23577117919921875 s
    • Merge
      • Pre change
        • Total space allocated: 412.4 KiB
        • Line specific space allocated (147): N/A
        • Merge runtime: 0.031603097915649414 s
      • Post change
        • Total space allocated: 412.4 KiB
        • Line specific space allocated (147): N/A
        • Merge runtime: 0.03347277641296387 s
  • 5000
    • Profile
      • Pre change
        • Total space allocated: 1.8 MiB
        • Line specific space allocated (147): 1005.7 KiB
        • Profile runtime: 0.45140695571899414 s
      • Post change
        • Total space allocated: 1.4 MiB
        • Line specific space allocated (147): N/A
        • Profile runtime: 0.5386018753051758 s
    • Merge
      • Pre change
        • Total space allocated: 806.6 KiB
        • Line specific space allocated (147): N/A
        • Merge runtime: 0.052163124084472656 s
      • Post change
        • Total space allocated: 806.6 KiB
        • Line specific space allocated (147):
        • Merge runtime: 0.05759596824645996 s
  • 7500
    • Profile
      • Pre change
        • Total space allocated: 2.3 MiB
        • Line specific space allocated (147): 1.4 MiB
        • Profile runtime: 0.6124231815338135 s
      • Post change
        • Total space allocated: 2.0 MiB
        • Line specific space allocated (147): N/A
        • Profile runtime: 0.6501669883728027 s
    • Merge
      • Pre change
        • Total space allocated: 1021.3 KiB
        • Line specific space allocated (147): N/A
        • Merge runtime: 0.05865597724914551 s
      • Post change
        • Total space allocated: 1021.3 KiB
        • Line specific space allocated (147): N/A
        • Merge runtime: 0.06494402885437012 s
  • 100000
    • Profile
      • Pre change
        • Total space allocated: 27.4 MiB
        • Line specific space allocated (147): N/A
        • Profile runtime: 6.970675230026245 s
      • Post change
        • Total space allocated: 27.4 MiB
        • Line specific space allocated (147): N/A
        • Profile runtime: 5.897289037704468 s
    • Merge
      • Pre change
        • Total space allocated: 8.1 MiB
        • Line specific space allocated (147): N/A
        • Merge runtime: 0.3647587299346924 s
      • Post change
        • Total space allocated: 8.1 MiB
        • Line specific space allocated (147): N/A
        • Merge runtime: 0.3217649459838867 s

Generated Dataset

  • 0
    • Profile
      • Pre change
        • Total space allocated: N/A
        • Line specific space allocated (147): N/A
        • Profile runtime: 0.024805068969726562 s
      • Post change
        • Total space allocated: N/A
        • Line specific space allocated (147): N/A
        • Profile runtime: 0.02689218521118164 s
    • Merge
      • Pre change
        • Total space allocated: N/A
        • Line specific space allocated (147): N/A
        • Merge runtime: .3828277587890625e-05 s
      • Post change
        • Total space allocated: N/A
        • Line specific space allocated (147): N/A
        • Merge runtime: 1.1920928955078125e-05 s
  • 100
    • Profile
      • Pre change
        • Total space allocated: 1.8 MiB
        • Line specific space allocated (147): 536.2 KiB
        • Profile runtime: 0.37052202224731445 s
      • Post change
        • Total space allocated: 1.5 MiB
        • Line specific space allocated (147): 1.5 KiB
        • Profile runtime: 0.37250804901123047 s
    • Merge
      • Pre change
        • Total space allocated: 890.8 KiB
        • Line specific space allocated (147): N/A
        • Merge runtime: 0.031355857849121094 s
      • Post change
        • Total space allocated: 890.8 KiB
        • Line specific space allocated (147): N/A
        • Merge runtime: 0.03441977500915527 s
  • 1000
    • Profile
      • Pre change
        • Total space allocated: 5.7 MiB
        • Line specific space allocated (147): 4.9 MiB
        • Profile runtime: 0.5443339347839355 s
      • Post change
        • Total space allocated: 1.5 MiB
        • Line specific space allocated (147): 1.5 KiB
        • Profile runtime: 0.4726378917694092 s
    • Merge
      • Pre change
        • Total space allocated: 613.2 KiB
        • Line specific space allocated (147): N/A
        • Merge runtime: 0.05542302131652832 s
      • Post change
        • Total space allocated: 613.2 KiB
        • Line specific space allocated (147):
        • Merge runtime: 0.0522770881652832 s
  • 5000
    • Profile
      • Pre change
        • Total space allocated: 27.2 MiB
        • Line specific space allocated (147): 25.6 MiB
        • Profile runtime: 1.3865718841552734 s
      • Post change
        • Total space allocated: 5.6 MiB
        • Line specific space allocated (147): N/A
        • Profile runtime: 1.2073299884796143 s
    • Merge
      • Pre change
        • Total space allocated: 2.1 MiB
        • Line specific space allocated (147): N/A
        • Merge runtime: 0.08409285545349121 s
      • Post change
        • Total space allocated: 2.1 KiB
        • Line specific space allocated (147):
        • Merge runtime: 0.08816409111022949 s
  • 7500
    • Profile
      • Pre change
        • Total space allocated: 38.9 MiB
        • Line specific space allocated (147): 36.4 MiB
        • Profile runtime: 1.8580598831176758 s
      • Post change
        • Total space allocated: 7.8 MiB
        • Line specific space allocated (147): N/A
        • Profile runtime: 1.6603739261627197 s
    • Merge
      • Pre change
        • Total space allocated: 2.6 MiB
        • Line specific space allocated (147): N/A
        • Merge runtime: 0.10225391387939453 s
      • Post change
        • Total space allocated: 2.6 MiB
        • Line specific space allocated (147): N/A
        • Merge runtime: .09444808959960938 s
  • 100000
    • Profile
      • Pre change
        • Total space allocated: 535.0 MiB
        • Line specific space allocated (147): 486.3 MiB
        • Profile runtime: 19.998097896575928 s
      • Post change
        • Total space allocated: 129.3 MiB
        • Line specific space allocated (147): N/A
        • Profile runtime: 18.45617413520813 s
    • Merge
      • Pre change
        • Total space allocated: 38.5 MiB
        • Line specific space allocated (147): N/A
        • Merge runtime: 0.8092191219329834 s
      • Post change
        • Total space allocated: 38.5 MiB
        • Line specific space allocated (147): N/A
        • Merge runtime: 0.8238198757171631 s

Findings:
This change is effective when dealing with large amounts of text and string columns. In the "In repo" dataset there are not
many examples of these data entry classes and as a result it does not benefit as much from the change as the generated dataset does.
(generated dataset is of all 7 data classes mentioned in the dataprofiler/tests/space_time_analysis/throughput-test-guidelines.md)

ksneab7
ksneab7 commented Apr 24, 2023
View reviewed changes
dataprofiler/profilers/text_column_profile.py
@@ -144,7 +144,7 @@ def _update_vocab(
:type subset_properties: dict
:return: None
"""
data_flat = list(itertools.chain(*data))
data_flat = set(itertools.chain(*data))
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actual change to code that is being investigated with the description

@JGSweets JGSweets enabled auto-merge (squash) April 24, 2023 17:26
taylorfturner
taylorfturner reviewed Apr 24, 2023
View reviewed changes
dataprofiler/tests/space_time_analysis/dataset_generation.py Outdated
Comment on lines 19 to 29
class NpEncoder(json.JSONEncoder):
def default(self, obj):
if isinstance(obj, np.integer):
return int(obj)
if isinstance(obj, np.floating):
return float(obj)
if isinstance(obj, np.ndarray):
return obj.tolist()
return super().default(obj)


Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what is this???

dataprofiler/tests/space_time_analysis/structured_space_time_analysis.py Outdated
@@ -236,7 +236,8 @@ def dp_space_time_analysis(
if not os.path.exists(os.path.dirname(path)):
os.makedirs(os.path.dirname(path))
with open(path, "w") as fp:
json.dump(profile_times, fp, indent=4)
json.dump(profile_times, fp, indent=4, cls=NpEncoder)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe call this NumpyEncoder?

@ksneab7 ksneab7 force-pushed the optimization_for_text_column_profile_ksneab branch from 7e794c2 to e451b9f Compare April 24, 2023 17:34
@JGSweets JGSweets merged commit c2cee9e into capitalone:main Apr 24, 2023
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@taylorfturner taylorfturner taylorfturner approved these changes

@tyfarnan tyfarnan tyfarnan approved these changes

@JGSweets JGSweets Awaiting requested review from JGSweets

@micdavis micdavis Awaiting requested review from micdavis

Assignees

@ksneab7 ksneab7

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
4 participants
@ksneab7 @taylorfturner @tyfarnan @JGSweets