sapp.pipeline.ParseError: Unexpected trace fragment

[ignatirabo]

While trying to analyze Pysa output with command sapp --database-name sapp.db analyze taint-output.json, I get the following output.

2023-01-03 14:07:22,447 [INFO] Parsing analysis output...
Traceback (most recent call last):
  File "/usr/local/bin/sapp", line 8, in <module>
    sys.exit(cli())
  File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/click/decorators.py", line 84, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/sapp/cli_lib.py", line 207, in analyze
    pipeline.run(analysis_output, summary_blob)
  File "/usr/local/lib/python3.7/dist-packages/sapp/pipeline/__init__.py", line 441, in run
    next_input, summary = step.run(next_input, summary)
  File "/usr/local/lib/python3.7/dist-packages/sapp/pipeline/base_parser.py", line 254, in run
    summary.get("old_linemap_file"),
  File "/usr/local/lib/python3.7/dist-packages/sapp/pipeline/base_parser.py", line 203, in analysis_output_to_dict_entries
    for typ, key, e in self._analysis_output_to_parsed_tuples(inputfile):
  File "/usr/local/lib/python3.7/dist-packages/sapp/pipeline/base_parser.py", line 141, in _analysis_output_to_parsed_tuples
    for e in entries:
  File "/usr/local/lib/python3.7/dist-packages/sapp/pipeline/pysa_taint_parser.py", line 99, in parse
    for entry in self.parse_handle(handle):
  File "/usr/local/lib/python3.7/dist-packages/sapp/pipeline/pysa_taint_parser.py", line 106, in parse_handle
    yield from self._parse_by_type(entry)
  File "/usr/local/lib/python3.7/dist-packages/sapp/pipeline/pysa_taint_parser.py", line 172, in _parse_by_type
    yield from self._parse_issue(entry["data"])
  File "/usr/local/lib/python3.7/dist-packages/sapp/pipeline/base_parser.py", line 87, in wrapper
    yield from func(self, json, *args)
  File "/usr/local/lib/python3.7/dist-packages/sapp/pipeline/pysa_taint_parser.py", line 239, in _parse_issue
    ) = self._parse_issue_traces(json["traces"], "backward", "sink")
  File "/usr/local/lib/python3.7/dist-packages/sapp/pipeline/pysa_taint_parser.py", line 298, in _parse_issue_traces
    return self._parse_issue_trace_fragments(leaf_port, trace["roots"])
  File "/usr/local/lib/python3.7/dist-packages/sapp/pipeline/pysa_taint_parser.py", line 311, in _parse_issue_trace_fragments
    for fragment in self._parse_trace_fragment(leaf_port, trace):
  File "/usr/local/lib/python3.7/dist-packages/sapp/pipeline/pysa_taint_parser.py", line 351, in _parse_trace_fragment
    yield from self._parse_trace_fragment_v3(leaf_port, trace)
  File "/usr/local/lib/python3.7/dist-packages/sapp/pipeline/pysa_taint_parser.py", line 468, in _parse_trace_fragment_v3
    raise ParseError("Unexpected trace fragment.", received=trace)
sapp.pipeline.ParseError: Unexpected trace fragment.
Received: `{'kinds': [{'leaves': [{'name': 'print'}], 'kind': 'Print'}], 'origin': {'filename': 'imprecision1.py', 'line': 5, 'start': 6, 'end': 22}}

Program being analyzed:

n: int = input("Input a number: ")
i: int = 'I am a number'
res = n+i
print(res.bit_length())

And the taint-output.json I want to analyze:

{"file_version":3,"config":{"repo":"/Users/tiraboschi/pysa_project"}}
{"kind":"issue","data":{"callable":"imprecision1.$toplevel","callable_line":1,"code":5002,"line":5,"start":6,"end":22,"filename":"imprecision1.py","message":"Data from [UserControlled] source(s) may reach [Print] sink(s)","traces":[{"name":"forward","roots":[{"kinds":[{"leaves":[{"name":"input"}],"kind":"UserControlled"}],"local_features":[{"always-via":"obscure:model"},{"always-type":"integer"},{"always-type":"scalar"},{"always-via":"tito"}],"tito_positions":[{"line":4,"start":6,"end":7},{"line":5,"start":6,"end":9}],"origin":{"filename":"imprecision1.py","line":2,"start":9,"end":34}}]},{"name":"backward","roots":[{"kinds":[{"leaves":[{"name":"print"}],"kind":"Print"}],"origin":{"filename":"imprecision1.py","line":5,"start":6,"end":22}}]}],"features":[{"always-via":"obscure:model"},{"always-type":"integer"},{"always-type":"scalar"},{"always-via":"tito"}],"sink_handle":{"kind":"Call","callee":"print","index":0,"parameter":"formal(*rest0)"},"master_handle":"imprecision1.$toplevel:5002:0:Call|print|0|formal(*rest0):f92c08fecfc0d6d0890769682e75edf0"}}
{"kind":"issue","data":{"callable":"imprecision2.convert","callable_line":7,"code":5002,"line":10,"start":10,"end":14,"filename":"imprecision2.py","message":"Data from [UserControlled] source(s) may reach [Print] sink(s)","traces":[{"name":"forward","roots":[{"kinds":[{"leaves":[{"name":"input"}],"kind":"UserControlled"}],"local_features":[{"always-via":"tito"},{"always-via":"obscure:unknown-callee"}],"tito_positions":[{"line":9,"start":20,"end":21}],"origin":{"filename":"imprecision2.py","line":8,"start":8,"end":33}}]},{"name":"backward","roots":[{"kinds":[{"leaves":[{"name":"print"}],"kind":"Print"}],"origin":{"filename":"imprecision2.py","line":10,"start":10,"end":14}}]}],"features":[{"always-via":"tito"},{"always-via":"obscure:unknown-callee"}],"sink_handle":{"kind":"Call","callee":"print","index":0,"parameter":"formal(*rest0)"},"master_handle":"imprecision2.convert:5002:0:Call|print|0|formal(*rest0):b730aacd59b2cd2e9376c74ea615a9bb"}}
{"kind":"model","data":{"callable":"imprecision2.get_zero","tito":[{"port":"formal(n)[__mul__]","taint":[{"kinds":[{"return_paths":{"":0},"kind":"LocalReturn"}],"local_features":[{"always-via":"obscure:unknown-callee"}],"tito":null}]}]}}
{"kind":"issue","data":{"callable":"imprecision3.convert","callable_line":7,"code":5002,"line":13,"start":14,"end":18,"filename":"imprecision3.py","message":"Data from [UserControlled] source(s) may reach [Print] sink(s)","traces":[{"name":"forward","roots":[{"kinds":[{"leaves":[{"name":"input"}],"kind":"UserControlled"}],"local_features":[{"always-via":"tito"},{"always-via":"obscure:unknown-callee"}],"tito_positions":[{"line":9,"start":20,"end":21}],"origin":{"filename":"imprecision3.py","line":8,"start":8,"end":33}}]},{"name":"backward","roots":[{"kinds":[{"leaves":[{"name":"print"}],"kind":"Print"}],"origin":{"filename":"imprecision3.py","line":13,"start":14,"end":18}}]}],"features":[{"always-via":"tito"},{"always-via":"obscure:unknown-callee"}],"sink_handle":{"kind":"Call","callee":"print","index":0,"parameter":"formal(*rest0)"},"master_handle":"imprecision3.convert:5002:0:Call|print|0|formal(*rest0):05da74f5000e93661778704068697bda"}}
{"kind":"model","data":{"callable":"imprecision3.get_zero","tito":[{"port":"formal(n)[__mul__]","taint":[{"kinds":[{"return_paths":{"":0},"kind":"LocalReturn"}],"local_features":[{"always-via":"obscure:unknown-callee"}],"tito":null}]}]}}
{"kind":"issue","data":{"callable":"imprecision4.convert","callable_line":7,"code":5002,"line":13,"start":14,"end":18,"filename":"imprecision4.py","message":"Data from [UserControlled] source(s) may reach [Print] sink(s)","traces":[{"name":"forward","roots":[{"kinds":[{"leaves":[{"name":"input"}],"kind":"UserControlled"}],"local_features":[{"always-via":"tito"},{"always-via":"obscure:unknown-callee"}],"tito_positions":[{"line":9,"start":20,"end":21}],"origin":{"filename":"imprecision4.py","line":8,"start":8,"end":33}}]},{"name":"backward","roots":[{"kinds":[{"leaves":[{"name":"print"}],"kind":"Print"}],"origin":{"filename":"imprecision4.py","line":13,"start":14,"end":18}}]}],"features":[{"always-via":"tito"},{"always-via":"obscure:unknown-callee"}],"sink_handle":{"kind":"Call","callee":"print","index":0,"parameter":"formal(*rest0)"},"master_handle":"imprecision4.convert:5002:0:Call|print|0|formal(*rest0):5d60f5a756115ff632602c48995522a3"}}
{"kind":"model","data":{"callable":"imprecision4.get_zero","tito":[{"port":"formal(n)[__mul__]","taint":[{"kinds":[{"return_paths":{"":0},"kind":"LocalReturn"}],"local_features":[{"always-via":"obscure:unknown-callee"}],"tito":null}]}]}}
{"kind":"model","data":{"callable":"input","sources":[{"port":"result","taint":[{"kinds":[{"kind":"UserControlled"}],"declaration":null}]}],"modes":["Obscure"]}}
{"kind":"model","data":{"callable":"os.system","sinks":[{"port":"formal(command)","taint":[{"kinds":[{"kind":"RemoteCodeExecution"}],"declaration":null}]}],"modes":["Obscure"]}}
{"kind":"model","data":{"callable":"print","sinks":[{"port":"formal(*rest0)","taint":[{"kinds":[{"kind":"Print"}],"declaration":null}]}],"modes":["Obscure"]}}
{"kind":"issue","data":{"callable":"source.convert","callable_line":7,"code":5001,"line":9,"start":22,"end":32,"filename":"source.py","message":"Data from [UserControlled] source(s) may reach [RemoteCodeExecution] sink(s)","traces":[{"name":"forward","roots":[{"kinds":[{"leaves":[{"name":"input"}],"kind":"UserControlled"}],"origin":{"filename":"source.py","line":8,"start":17,"end":38}}]},{"name":"backward","roots":[{"kinds":[{"features":[{"always-via":"format-string"},{"always-via":"tito"}],"leaves":[{"name":"os.system"}],"length":1,"kind":"RemoteCodeExecution"}],"call":{"position":{"filename":"source.py","line":9,"start":22,"end":32},"resolves_to":["source.get_image"],"port":"formal(url)"}}]}],"features":[{"always-via":"format-string"},{"always-via":"tito"}],"sink_handle":{"kind":"Call","callee":"source.get_image","index":0,"parameter":"formal(url)"},"master_handle":"source.convert:5001:0:Call|source.get_image|0|formal(url):71e3ce573ab7da30d5a55d88882319a9"}}
{"kind":"model","data":{"callable":"source.get_image","sinks":[{"port":"formal(url)","taint":[{"kinds":[{"leaves":[{"name":"os.system"}],"kind":"RemoteCodeExecution"}],"local_features":[{"always-via":"format-string"},{"always-via":"tito"}],"tito_positions":[{"line":4,"start":40,"end":43}],"origin":{"filename":"source.py","line":5,"start":21,"end":28}}]}],"tito":[{"port":"formal(url)","taint":[{"kinds":[{"return_paths":{"":0},"length":1,"kind":"LocalReturn"}],"local_features":[{"always-via":"obscure:model"},{"always-type":"integer"},{"always-via":"format-string"},{"always-type":"scalar"},{"always-via":"tito"}],"tito_positions":[{"line":4,"start":40,"end":43},{"line":5,"start":21,"end":28}],"tito":null}]}]}}
{"kind":"model","data":{"callable":"pstats.FunctionProfile.__init__","tito":[{"port":"formal(tottime)","taint":[{"kinds":[{"return_paths":{"[tottime]":999999},"kind":"LocalReturn"}],"tito":null}]},{"port":"formal(percall_tottime)","taint":[{"kinds":[{"return_paths":{"[percall_tottime]":999999},"kind":"LocalReturn"}],"tito":null}]},{"port":"formal(percall_cumtime)","taint":[{"kinds":[{"return_paths":{"[percall_cumtime]":999999},"kind":"LocalReturn"}],"tito":null}]},{"port":"formal(ncalls)","taint":[{"kinds":[{"return_paths":{"[ncalls]":999999},"kind":"LocalReturn"}],"tito":null}]},{"port":"formal(line_number)","taint":[{"kinds":[{"return_paths":{"[line_number]":999999},"kind":"LocalReturn"}],"tito":null}]},{"port":"formal(file_name)","taint":[{"kinds":[{"return_paths":{"[file_name]":999999},"kind":"LocalReturn"}],"tito":null}]},{"port":"formal(cumtime)","taint":[{"kinds":[{"return_paths":{"[cumtime]":999999},"kind":"LocalReturn"}],"tito":null}]}]}}
{"kind":"model","data":{"callable":"pstats.StatsProfile.__init__","tito":[{"port":"formal(total_tt)","taint":[{"kinds":[{"return_paths":{"[total_tt]":999999},"kind":"LocalReturn"}],"tito":null}]},{"port":"formal(func_profiles)","taint":[{"kinds":[{"return_paths":{"[func_profiles]":999999},"kind":"LocalReturn"}],"tito":null}]}]}}

I don't understand why I am getting this error: is it my fault or some incompatibility?

[arthaud]

I think we probably made non-backward compatible changes in our output format recently.
What version of pyre-check and fb-sapp are you using? We might need to push a new version for one of them.

[ignatirabo]

I checked on PIP. These are the versions:

fb-sapp             0.5.1
pyre-check          0.9.17
pyre-extensions     0.0.30

And thank you!

[arthaud]

The version of fb-sapp you are using is very outdated (from 2021), could you try upgrading it?

[ignatirabo]

Hello, indeed there is versioning problem, by having an updated SAPP the problem goes away.
The problem itself arises from the fact that pyre-check 0.9.17 depends on pyre-extensions>=0.0.29 but fb-sapp 0.5.4 depends on pyre-extensions==0.0.27.
Basically by installing them together with pip install pyre-check fb-sapp or pip install fb-sapp pyre-check one of both doesn't work.

For now the solution for me is having two virtual environments, one for Pyre/Pysa, and one for SAPP.

Thank you for the help!