Repository navigation
v0.0.4
·
133 commits
to main
since this release
In short
- New: kflow: Go to oops/WARN line (also
python3 -m kflow <kernel> --oops <log>). Paste aWARNING:orkernel BUG atline, a whole oops or syzbot report,func+0xofforfile:line, and kflow goes to that BUG/WARN site; when the log contradicts the site at that address (for example a log from another build), it saysmismatchand goes nowhere. - New, experimental (x86_64, off by default): kflow: Let WARN sites fall through (experimental). IDA ends code flow at a WARN
ud2; this adds the flow after it, so the code after the trap becomes part of its function. - Safer Annotate: no references from bytes that are not instructions yet (Annotate during auto-analysis), and a static-key target or fixup inside another function gets a data reference instead of a jump.
- Pseudocode comments also appear when the decompiler loads after kflow;
tools/kflow_headless.py --jsonexports every decoded site; the README starts with a Features list. - Tested on real kernel files with fake
ida_*modules; not yet run in real IDA.
Details
- Annotate run while IDA's auto-analysis is still going no longer adds references from sites that are not instructions yet. The IDA SDK says the source of a cross-reference must be an item head and that
get_item_end()counts unexplored bytes as 1-byte items, so on a freshly loaded kernel kflow's "the fixup is the next instruction" test could not match: such exception-table entries (common on arm64 from 5.16 and at x86's plainrdmsr/wrmsr,EX_TYPE_RDMSR/WRMSR) got a user jump reference that a later Annotate kept. Now a site that is not code (ida_bytes.is_code(ida_bytes.get_flags(ea))) gets its comment but no jump, data or static-key reference, and Annotate reports how many such sites there are and says to run it again after auto-analysis (alsotools/kflow_headless.py --no-analysis). A later Annotate deletes a reference to the next instruction that an earlier one recorded (also one added by an older kflow), and reports it. This follows from reading the IDA SDK (ida_xref,ida_bytesstubs), not from a run in IDA; tested with fakeida_*modules only, which now model instruction heads (get_flags/is_code), real instruction ends where known and no functions while analysis runs. Found by a pre-release review of the IDA calls.tests/check_ida_api.pyalso checksida_bytes.get_flagsandis_code. When that later Annotate (or Remove) deletes such a reference and IDA's ordinary flow to the next instruction is gone afterwards (the SDK does not say whetherdel_cref()clears it), kflow puts the ordinary flow back (fl_F). On a database whose auto-analysis has finished, sites that are still not code are reported as "not code in IDA" with a hint to press C there, instead of "run Annotate again when auto-analysis has finished". - Pseudocode comments when the Hex-Rays decompiler loads after kflow: kflow tried its pseudocode hook only when it loaded (and on Annotate), so a database annotated earlier showed no comments until Annotate ran again. IDAPython initialises its decompiler bindings when the decompiler plugin loads (
ui_plugin_loadedin IDAPython'spy_hexrays.hpp), so when the decompiler is not there at kflow's start, kflow now installs aida_kernwin.UI_Hookswhoseplugin_loadedtries again until the hook is in, and removes it when kflow unloads. Whether IDA loads the decompiler before or after kflow was not checked. This follows from reading the IDA SDK; tested with fake modules only (a fake decompiler that is missing at kflow's init and present later).tests/check_ida_api.py: 205 checks, 0 missing;tests/check_ida_names.py: 86 names, 0 missing in the stubs of IDA 9.1 to 9.5. - New action kflow: Go to oops/WARN line and
python3 -m kflow <kernel> --oops <log>(-for stdin;--jsonwrites the match): paste aWARNING:,kernel BUG at,RIP:orpc :line, a whole oops or syzbot report,func+0xofforfile:line. kflow takes the function+offset (from the report's ownRIP:/pc :line first, then theWARNING:orkernel BUG atline) and goes to the BUG/WARN site at exactly that address (a name the log prints without a Clang LTO suffix,fooforfoo.123,foo.llvm.Norfoo.cfi, tries all those functions, the plain one too; GCC'sfoo.part.N,.isra.N,.constprop.Nand.coldare printed in full, sofoo+0x10never means them; the output names the function used); if there is none it says so, goes to the log's address and lists that function's sites (those at the log'sfile:linefirst, then nearest first; the function as this file has it, whatever size the log prints (in IDA up to the end of its last chunk before the next name: code after a gap, such as after a WARNud2, can be a tail chunk), and an offset past its end, from another build, lists them without going anywhere, also when the next function has a site at that address); with only afile:lineshared by several sites it lists just those (in IDA: the BUG/WARN chooser with only those rows). When the log contradicts the site at its address (itsWARNING:/kernel BUG atline names anotherfile:lineor kind; or that line has nofile:line(a kernel withoutCONFIG_DEBUG_BUGVERBOSE) and its function+offset names another address than theRIP:/pc :line (Kernel BUG at %pBnames a BUG at a function's first byte as the previous symbol plus its size: the same address); or, without aWARNING:/kernel BUG atline that names the site, it prints another function size than this file's, kallsyms' size to the next symbol counted: a log from another build, or two reports pasted together) kflow saysmismatch, exits 1 and goes nowhere; it lists the sites at the log'sfile:line, else the site at that address (in IDA: a warning and the chooser). The same when the log names a function+offset this file does not have and itsfile:linehas one site here (that line's site may be another one). A paste with several reports is cut into them (atcut here/end trace, the nextWARNING:/kernel BUG atline, or another report's first line after a register dump; not at syzbot's repeatedWARNING:line: syzbot prints aWARNING:orRIP:line once for each inlined frame,func file:line [inline], before the one with the function+offset, which is the one used; not atKernel panic - not syncing:: that line and the stack dump after it belong to the report in progress, so apanic_on_warnreport, whose panic dump on x86 shows the trap'sRIP:line again, counts as one; not at x86's executive summary afterend trace, the trap'sRIP:line and registers printed again, which belongs to the report before it); the first one with aWARNING:/kernel BUG atline or a trapRIP:/pc :line is used, and a note says how many there are. ARIP:/pc :line after the report's call trace began (x86 prints an interrupted context's registers where the trace crosses an IRQ or exception entry, as in a KASAN report from an interrupt) or afterNMI backtrace for cpu N(another CPU's registers, in hung-task and RCU-stall reports) is not the report's trap, nor is a call-trace entry that repeats it (arm64 repeats the pc there). The trap's own lines after the trace are used, so the bottom of a report pasted alone still finds its trap: the executive summary of Linux 4.9 (RIP [<%p>] %pS) and of 4.10 to 4.16 (RIP: %pS RSP: %016lx), and theRIP:right after the#UDentry frameasm_exc_invalid_opoutside an NMI backtrace (panic()'s dump underpanic_on_warn). A report with noWARNING:/kernel BUG atline and no trapRIP:/pc :line (KASAN, lockdep, hung task) or a pasted call trace of two or more entries counts only if its first reliable entry is itself a BUG/WARN site (that entry is a caller, not a trap); otherwise kflow says there is no BUG/WARN trap in it (naming such aRIP:/pc :line, if there is one) and exits 1;?entries are never used.--oopswith--listor--limitis an error (exit 2); a module file namedfoo.ko.debug(Fedora / RHEL debuginfo) is modulefoo; in IDA a jump that fails (an address outside the database) gives a warning. On the demo kernelfile:linealone matched four sites atmm/userfaultfd.c:755;mfill_atomic_copy+0x674picks0xffffffff8e629bc4. The line formats are taken from the kernel's printk calls (__warn()up to 6.18 and from 6.19,report_bug(), x86show_ip()and call traces, arm64__show_regs(),%pSwith[module], the 4.9 forms, 2.6 / 3.xWARNING: at file:line func+0xoff()andKernel BUG at <address>) and from syzbot's reports (the source line syzbot appends after the symbol of aWARNING:/Kernel BUG atline is not taken as that line'sfile:line: for an inlined trap it is the outer frame's); the prefixes ofdmesg(also-T,-H,-d,-x,--time-format iso),/dev/kmsg,journalctl(short,-precise,-iso,-monotonic) andlogcat -b kernel(its time and level; logd makes the first word of apc :,lr :orRIP:line the tag, so from logcat such lines are not recognised aspc :/RIP:lines) are removed, also before call-trace entries.tests/test_oops.py: log samples built from those formats; on each test kernel and module, aRIP:/pc :line made for every BUG/WARN site inside a function finds that site (for example all 24,386 on Android GKI 5.10; same-named static functions with a site at the same offset are listed as candidates; the 231 sites there in functions with a Clang LTO suffix also from the name the kernel prints,cpu_replace_ttbr1forcpu_replace_ttbr1.989: found, or listed among the copies of that name), also with the site's ownWARNING:/kernel BUG atline before it, also with and from the line a kernel withoutCONFIG_DEBUG_BUGVERBOSEprints for it (WARNING: ... at %pS,Kernel BUG at %pB), and with another line or kind is a mismatch (among same-named functions: listed). The IDA action is tested only with fakeida_*modules (also a site in a later chunk of its function); not yet run in real IDA. In IDA only the function IDA shows under the log's name is tried (IDA names one function so); for a static function a note says that another one of that name is not tried, unless the log'sWARNING:/kernel BUG atline confirms the site (a function whose name is neither public nor weak in IDA,ida_name.is_public_name/is_weak_name). When the site found in such a static function is amismatch, the warning and a note also name that cause:or IDA shows another static function of that name (IDA names only one).tests/check_ida_api.py: 205 checks, 0 missing. tests/check_ida_names.py <stub dir>...: checks that everyida_*name kflow uses exists in a folder of IDAPython stubs; all 86 are in the stubs of IDA 9.1 to 9.5 (PyPIidapro0.0.5, 0.0.7, 0.0.9 and 0.0.11, and the IDA 9.5 stubs thattests/check_ida_api.pyis run against).- Annotate: a static-key target or fixup inside another IDA function (
ida_funcs.get_func; not at its start, not in a tail the source's function shares, not the source function's own.coldpart) now gets a user data reference (dr_O) instead of a jump reference (fl_JN), so that IDA is not asked to treat the other function's code as flow of this one (it could try to attach it as a tail chunk); other targets get the jump reference as before. This needs IDA's functions: Annotate run while auto-analysis is still going says to run it again afterwards, and a later Annotate changes references kflow added earlier (also by older versions) between the two kinds to match IDA's current functions. Recorded and removed like the others; counts reported. Tested only on fakeida_*modules; not yet run in real IDA. tools/kflow_headless.py --json PATH: writes every decoded BUG/WARN, static-key and exception-table site with IDA's function names (addresses as hex strings; static-key records saylikely/unlikelyand whether the site is a jump now), read before Annotate or Remove change anything. Tested on fakeidapro/ida_*modules only.- README: a Features section up front (each feature links to where it is described); the new WARN fall-through actions are described in
docs/warnflow.md; the kernel-module details moved todocs/modules.mdand the full Limits text todocs/limits.md; Limits shortened to one line per limit (the line saying kflow does not change how IDA treats a WARNud2is replaced by one for WARN fall-through), and long lines split into bullets; no facts changed except that the freed-init limit now covers all three tables. - Experimental, off by default, x86_64 only: kflow: Let WARN sites fall through (experimental) and kflow: Remove WARN fall-through. IDA ends code flow at
ud2, so the code after a WARN trap is unreachable and missing from the function. The action adds ordinary flow after exactly the WARNud2traps of the kernel's__bug_tablethrough a processor hook (BUG traps are left alone), lets IDA reanalyse them, appends newly reachable code that is in no function to the trap's function as a tail (never code of another function), and with Hex-Rays installs a microcode filter that decompiles those traps as a call to__WARN()that returns (unconfirmed whether Hex-Rays accepts it forud2; if it refuses, the rest stays). The sites are recorded in the database as offsets from the image base; the hook is installed only for databases with that record, again whenever kflow loads one, and removed when kflow unloads. Remove takes out the tails, the flow and the record. Not on arm64: public Hex-Rays output of arm64 kernels shows code continuing afterbrk #0x800(IDA version unknown); on an arm64 database the action changes nothing and reports whether IDA's ARM module marksbrkas the end of flow. Prior art credited in the README: Jason A. Donenfeld'slinux_und.py. Not yet run in real IDA; tested only on fakeida_*modules (tests/test_warnflow_stub.py: on the demo kernel flow after all 7,504 WARNud2traps and none of the 1,825 BUGud2traps, i.e. the 1,825 of its 2,019 BUG entries whose trap is aud2in code).- The decompiler filter is installed only when at least one site got flow (if IDA ignored the hook, the action warns and installs nothing for the decompiler). Each function tail kflow appends is recorded with its end, and Remove takes out only a tail chunk that still starts and ends there and belongs to the site's function; a tail IDA merged with its own chunks (unknown whether IDA does that) stays and is counted.
Built by an AI agent (Claude, operated by Nightshift Labs). Not yet run in real IDA.