Update Metadata type in capa main

[Aayush-Goel-04]

Closes Issue #1411
Closes PR #1444

Checklist

• No CHANGELOG update needed

• No new tests needed

• No documentation update needed

[Aayush-Goel-04]

I have made some temporary changes here, Will add from_capa apis to other Models and update it soon.
IDA also used meta as dict, so changed that too.

[Aayush-Goel-04]

capa/capa/main.py

Lines 243 to 255 in 8d016de

	def find_capabilities(ruleset: RuleSet, extractor: FeatureExtractor, disable_progress=None) -> Tuple[MatchResults, Any]:
	all_function_matches = collections.defaultdict(list) # type: MatchResults
	all_bb_matches = collections.defaultdict(list) # type: MatchResults
	all_insn_matches = collections.defaultdict(list) # type: MatchResults
	meta = {
	"feature_counts": {
	"file": 0,
	"functions": {},
	},
	"library_functions": {},
	} # type: Dict[str, Any]

Here should I change feature_counts & library_functions dict to their Respective models mentioned in ResultDocument. Then we won't have to recast meta in dict and then to Metadata. It will be easy to update any changes.

[williballenthin]

would you mark this PR as a draft until you're ready for review?

[Aayush-Goel-04]

I have looked into meta types used in main, direct partial update to meta data with outputs from find_capabilities and compute_layout is not possible since froze = True for FrozenModel Types. So where ever we want to update metadata we have do it via __dict__.update(data).

[williballenthin]

Would it make sense to not make these types Frozen then? I think I generally prefer frozen when it’s a choice, but since we’re not hashing the metadata object then I don’t think there’s a good reason for it to be frozen.

[Aayush-Goel-04]

I don’t think there’s a good reason for it to be frozen.

If Frozen = True, we can't do direct item assignment, here, for example using:

meta.analysis.features_counts = counts["feature_counts"]

for the changes I made in this PR shows error:

File "pydantic/main.py", line 359, in pydantic.main.BaseModel.__setattr__
TypeError: "Analysis" is immutable and does not support item assignment

thus, it is updated via __dict__.update(counts).
@williballenthin If frozen is removed we won't need to update it via dict.

[williballenthin]

lets remove frozen so that we can do the updates more easily

[Aayush-Goel-04]

Then we won't need from_capa functionality for most of the classes in result_document since we can easily update meta.

[Aayush-Goel-04]

@williballenthin if we are removing frozen, I think we should change name FrozenModel class, as it can be misleading. Can you suggest what I should name it.

[Aayush-Goel-04]

Also can you look into script capa_as_library.py , since we have set meta format as MetaData everywhere. Should the output format in script as Dictionary be removed.

[williballenthin]

ideally we can just use the rdoc.Metadata constructor directly here.

[williballenthin]

ideally we can just use the Metadata constructor directly here.

[williballenthin]

you're doing good work @Aayush-Goel-04, keep it up! thank you!

[williballenthin]

Suggested change

# TODO: #1411 use a real type, not a dict here.

[williballenthin]

lets not call this Frozen if its not frozen.

for things that can still stay frozen, lets use the existing FrozenModel. but lets also introduce a Model class for things that aren't frozen.

[williballenthin]

these changes look awesome! the only thing i'd like to see is the new Model class, then lets merge and get on to the next thing! thanks @Aayush-Goel-04!

[williballenthin]

lgtm
🎉

[Aayush-Goel-04]

Hey @williballenthin @mr-tz ,
I am really interested in malware analysis / security. Can you recommend me some good issues to work on.

Thanks

[mr-tz]

Awesome, some issues that come to mind:

• add profiling hooks to show how long it takes to match each function #1290 (related: Performance and accuracy: filter out functions #1286)
• flirt: consider using rizin/sigdb signatures #1094
• show the prevalence of rules in the output #520 (we can provide a [test] prevalence database for this)
• new idea: review the capa output rendering and draft/implement improvements

I think these may be a good fit for your interests and skill set! Happy to hear your thoughts on those or any other open issue. We really appreciate your interest and all the great contributions!