fix: let a callback call igraph again

[krlmlr]

This PR was prepared with Claude Code (Claude Opus).

Fixes #253. Found while working on #2845, where a deprecated function used as a callback turned out to take the session down.

The bug

Every R callback igraph invokes from C could crash. bfs() is the shortest reproducer, but it is not specific to it:

g <- make_tree(10)
bfs(g, root = 1, callback = function(graph, data, extra) { print(graph); FALSE })
#> Error: At core/dqueue.pmt:120 : Assertion failed: q->stor_begin != NULL.

That is #253, open since 2018. It needs no intent to reproduce: length() on a graph calls vcount(), and lifecycle gets there through rlang::trace_back() while it assembles a backtrace, so a deprecated function used as a callback was enough:

bfs(g, root = 1, callback = function(...) { is.igraph(g); FALSE })

Every callback family is affected, and several take R down outright rather than erroring:

• bfs(), dfs() — igraph assertion
• cliques(), max_cliques(), motifs(), simple_cycles() — igraph assertion
• isomorphisms(), subgraph_isomorphisms(), arpack() — R aborts
• cluster_leading_eigen() — R aborts
• attribute combination functions, so simplify(), contract() and the operators — igraph assertion, or R aborts

Why

R_igraph_finalizer() calls IGRAPH_FINALLY_FREE(), and every igraph function calls it from on.exit() to clean up after an algorithm that was left half-way. An igraph function reached from a callback is not that case: the "finally" stack then holds the structures of the algorithm that is still running, and freeing them leaves it working on destroyed memory. The algorithm carries on and trips over an assertion, or writes into freed memory and takes R with it.

The chain for the lifecycle case, from a traced .igraph.progress():

rlang::trace_back() → vec_ptype_shape() → dim2() → length() → length.igraph()
                    → vcount() → vcount_impl() → on.exit(.Call(Rx_igraph_finalizer))
                    → IGRAPH_FINALLY_FREE()   # frees the BFS queue

The fix

The finalizer stands aside while a callback is running. Rx_igraph_eval_callback() counts the callbacks in flight and every path that hands R code to a running algorithm goes through it: the search callbacks in rcallback.c, Rx_igraph_safe_eval_in_env() for bfs(), dfs() and the attribute combination functions, and the ARPACK multiplication.

The count is maintained with R_UnwindProtect() rather than a plain decrement afterwards. A callback that raises an error unwinds past the eval, and a count left standing would silence the finalizer for the rest of the session — which is worse than the original bug, because cleanup after a genuine error then stops happening too. I know because that is what my first attempt did: it counted running algorithms around IGRAPH_R_CHECK, and the leak turned into free(): invalid size several test files later.

Nothing else changes. The algorithm cleans up after itself when it finishes, and the error path in Rx_igraph_handle_safe_eval_result_in_env() still calls IGRAPH_FINALLY_FREE() before it aborts — that one runs after the callback has returned, so it is outside the guard.

This is narrower than #2548, which would give the R glue a re-entrant IGRAPH_FINALLY API with local stacks. That is still the thorough answer; this one stops the crashes without touching the API or the generated code.

Testing

New tests/testthat/test-nested-igraph-calls.R covers a nested igraph call from every callback family listed above, the deprecation case that led here, the reproducer from #253, and that an error raised in a callback still reaches the caller.

The bfs() test that covered this is no longer skipped:

 test_that("BFS callback does not blow up when another igraph function is raised within the callback", {
-  skip("nested igraph call handling not implemented yet")

Full suite passes; the two failures seen locally are environmental (a blocked download in test-foreign.R, and a callr subprocess in test-other.R that needs an installed igraph).

• By submitting this pull request, I assign the copyright of my contribution to The igraph development team.

🤖 Generated with Claude Code

https://claude.ai/code/session_01WJx1uH2GvN5DcqdVzDLvwW

---

Generated by Claude Code