Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update mde_proc_pub.pkl #709

Merged
merged 2 commits into from Sep 5, 2023
Merged

Conversation

FlorianBracq
Copy link
Collaborator

@FlorianBracq FlorianBracq commented Sep 5, 2023

With the latest version of pandas (2.1.0), some pickle files could not be parsed with the function read_pickle:

  • tests/testdata/mde_proc_pub.pkl
  • tests/testdata/mordor/mitre_tech_cache.pkl
  • tests/testdata/mordor/mitre_tact_cache.pkl

The error triggered in tests/transform/test_process_tree_utils.py and led to all the unit tests in this module to fail.

I'm not sure exactly how those files were generated, but to fix the issue I applied the following steps:

  • downgrade to pandas 2.0.3
  • Put a breakpoint on where the read_pickle function was called
  • Run the test test_process_tree_utils in debug mode
  • At the breakpoint, read the pickle file and through the debug console export it with the command:
    df.to_pickle(<filename>)

Then I upgraded back to 2.1.0 and the file appears to be properly parsed now.

@ianhelle ianhelle merged commit 44f3ec3 into microsoft:main Sep 5, 2023
17 checks passed
@ianhelle
Copy link
Contributor

ianhelle commented Sep 5, 2023

Awesome Florian - thanks for fixing. Not sure of the reason for this but presumably something to do with a subtle change in the pandas/pickle object model. in v2.1x. I think this version doesn't support Py3.8, which is why this build continued to work.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants