Skip to content

Add hooks, fickling.load(), and a JSON output format for usability #79

Merged
suhacker1 merged 34 commits intomasterfrom
sh/usability
Dec 19, 2023
Merged

Add hooks, fickling.load(), and a JSON output format for usability #79
suhacker1 merged 34 commits intomasterfrom
sh/usability

Conversation

@suhacker1
Copy link
Copy Markdown
Contributor

@suhacker1 suhacker1 commented Dec 15, 2023

This PR makes multiple feature additions for usability. Specifically, this PR adds a fickling import hook, global function hook, fickling.load() function, and a JSON output format for the check_safety component of the CLI. Each of these features can make it easier to integrate fickling into different codebases and tools.

This PR also updates the examples and tests to reflect these new features. Additional important changes made include:

  • Syncing is_likely_safe in fickle.py with check_safety in analysis.py: A new check_safety method was added to fickle.py as a wrapper with is_likely_safe being marked for deprecation.
  • Refactoring analysis.py: Not only was ProtoAnalysis split for simplicity, but more structure was added throughout the different analysis classes to enable the reporting of detailed results.
  • Adding new methods to pytorch.py: The PyTorchModelWrapper class now reports the identified file formats from the validation method.

I would especially appreciate feedback on:

  • Whether the hook tailored for torch.load should be included
  • Whether the naming throughout is consistent and appropriate
  • How usable the interfaces now exposed by fickling are
  • How useful the current state of the JSON output is

Example JSON Output ("test_unused_variables.json"):

{
    "severity": "OVERTLY_MALICIOUS",
    "analysis": "Call to `eval(b'[5, 6, 7, 8]')` is almost certainly evidence of a malicious pickle file\nVariable `_var0` is assigned value `eval(b'[5, 6, 7, 8]')` but unused afterward; this is suspicious and indicative of a malicious pickle file",
    "detailed_results": {
        "AnalysisResult": {
            "OvertlyBadEval": "eval(b'[5, 6, 7, 8]')",
            "UnusedVariables": [
                "_var0",
                "eval(b'[5, 6, 7, 8]')"
            ]
        }
    }
}

@suhacker1 suhacker1 marked this pull request as ready for review December 15, 2023 18:32
@suhacker1 suhacker1 requested a review from ESultanik as a code owner December 15, 2023 18:32
@suhacker1 suhacker1 marked this pull request as draft December 15, 2023 18:32
@suhacker1 suhacker1 changed the title Add hooks for usability Add hooks, `fickling.load(), and a JSON output format for usability Dec 18, 2023
@suhacker1 suhacker1 changed the title Add hooks, `fickling.load(), and a JSON output format for usability Add hooks, fickling.load(), and a JSON output format for usability Dec 18, 2023
Copy link
Copy Markdown
Contributor

@Boyan-MILANOV Boyan-MILANOV left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@suhacker1 good job on this! It needs some restructuring but the core idea is there :) I've left a bunch of comments, some are nits, but most of them are about code and architecture. Let's address all of them and then I'll make a second pass on the PR.

@suhacker1 suhacker1 marked this pull request as ready for review December 18, 2023 20:03
@suhacker1
Copy link
Copy Markdown
Contributor Author

suhacker1 commented Dec 19, 2023

Note: We decided not to include the PyTorch global hook in this PR. We also decided to remove the UNKNOWN severity type as we felt it was redundant.

@suhacker1
Copy link
Copy Markdown
Contributor Author

@Boyan-MILANOV This is ready for another review!

Copy link
Copy Markdown
Contributor

@Boyan-MILANOV Boyan-MILANOV left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM now! Good job! 🚀

@suhacker1 suhacker1 merged commit 08f98f2 into master Dec 19, 2023
@suhacker1 suhacker1 deleted the sh/usability branch January 4, 2024 03:57
dguido added a commit that referenced this pull request Feb 20, 2026
Renames AGENTS.md to CLAUDE.md and expands it from vulnerability
reporting guidance into full project context: commands, architecture,
design rules, testing patterns, code style, and CI. Design rules
distilled from PR review history (#79, #87, #195, #207, #210, #220).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
thomas-chauchefoin-tob pushed a commit that referenced this pull request Feb 20, 2026
Renames AGENTS.md to CLAUDE.md and expands it from vulnerability
reporting guidance into full project context: commands, architecture,
design rules, testing patterns, code style, and CI. Design rules
distilled from PR review history (#79, #87, #195, #207, #210, #220).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants