Python Extractor Upgrades — OOP & Decorator Semantics

[zaebee]

🎯 Feature Design: Python Extractor Upgrades — OOP & Decorator Semantics

1. Context & Problem Statement

Currently, our PythonExtractor extracts classes, methods, and functions as flat structures. However, production-grade Python code heavily utilizes:

1. Inheritance: class IngestionPipeline(BasePipeline)
2. Abstract Classes (ABC) & Abstract Methods: class BaseExtractor(ABC) with @abstractmethod
3. Decorators: @mcp.tool(), @app.command()

Without extracting these:

• We cannot resolve inherited method calls (e.g., child.method() where method is defined only in the parent class).
• We miss crucial framework registration entrypoints (like decorators registering CLI commands or MCP tools).

2. Data Contract & Ontology Upgrades

A. New Edge Type

We introduce a new structural edge type to represent class inheritance:

• EdgeType.EXTENDS (Source: Subclass CLASS, Target: Superclass CLASS FQN)

B. Node Metadata Additions

We enrich the Node metadata to store OOP and decorator states.

# Pseudo-schema inside src/cgis/core/models.py
class NodeMetadata(TypedDict, total=False):
    # For Classes / Methods:
    is_abstract: bool  # True if class inherits from ABC or method has @abstractmethod
    
    # For Functions / Methods:
    decorators: List[str]  # List of FQNs of applied decorators, e.g. ["mcp.tool", "app.command"]

---

3. Algorithmic Blueprint

Part A: AST Extraction (python_extractor.py)

We must update _walk to parse superclasses and decorator nodes in Tree-sitter [INDEX_3].

# Pseudo-code for python_extractor.py

def _process_class_inheritance(self, node: BaseNode, class_fqn: str, edges: List[Edge]):
    """
    Extracts base classes from 'class_definition' arguments.
    Example: class IngestionPipeline(BasePipeline):
    """
    superclasses_node = node.child_by_field_name("superclasses")
    if superclasses_node:
        # superclasses_node is an argument_list in tree-sitter-python
        for child in superclasses_node.children:
            if child.type in ("identifier", "attribute"):
                superclass_name = self._get_identifier(child)
                # Emit EXTENDS edge
                edges.append(Edge(
                    id=f"{class_fqn}_extends_{superclass_name}",
                    source=class_fqn,
                    # Target is a raw FQN name; Resolver will link it to actual Class Node
                    target=f"raw_class:{superclass_name}", 
                    type=EdgeType.EXTENDS
                ))

def _process_decorators(self, node: BaseNode, symbol_fqn: str, symbol_node: Node, edges: List[Edge]):
    """
    Parses decorator nodes.
    Example: @mcp.tool(name="foo")
    """
    decorators_node = node.child_by_field_name("decorators")
    if decorators_node:
        for child in decorators_node.children:
            if child.type == "decorator":
                # A decorator can be a 'call' (e.g. @mcp.tool()) or 'identifier' (e.g. @abstractmethod)
                nested = child.children[1] # Skip the '@' token
                
                deco_name = self._get_identifier(nested)
                symbol_node.metadata.setdefault("decorators", []).append(deco_name)
                
                # High-Thinking: Decorator execution is a functional dependency!
                # We emit a CALLS edge from the symbol to the decorator FQN.
                edges.append(Edge(
                    id=f"{symbol_fqn}_deco_{deco_name}",
                    source=symbol_fqn,
                    target=f"raw_call:{deco_name}",
                    type=EdgeType.CALLS,
                    confidence=0.5
                ))

Part B: Upgraded Inherited Method Resolution (resolver/engine.py)

This is the key compiler-grade logic. If a method call cannot be found on ChildClass, we must traverse up the EXTENDS edges to find the implementation in ParentClass [INDEX_3]!

# Pseudo-code inside ResolverEngine

def _resolve_instance_call(self, source_fqn: str, var_name: str, method_name: str) -> Optional[str]:
    # 1. Get class FQN of the variable (e.g., 'src.api.handlers.MyUserService')
    class_fqn = self._get_variable_class_fqn(source_fqn, var_name)
    if not class_fqn:
        return None

    # 2. Attempt to resolve on the class directly
    resolved = self._resolve_method_on_class_hierarchy(class_fqn, method_name)
    return resolved

def _resolve_method_on_class_hierarchy(self, class_fqn: str, method_name: str, visited: Set[str] = None) -> Optional[str]:
    """
    Recursively traverses up the EXTENDS hierarchy to resolve inherited methods.
    Prevents circular inheritance loops.
    """
    if visited is None:
        visited = set()
    if class_fqn in visited:
        return None
    visited.add(class_fqn)

    # 1. Check if the method is defined directly in this class
    if class_fqn in self._class_methods and method_name in self._class_methods[class_fqn]:
        return self._class_methods[class_fqn][method_name]

    # 2. If not found, look up EXTENDS edges pointing from this class
    # (Assuming we indexed extends edges as self._inheritance_tree: Dict[class_fqn, List[parent_fqn]])
    parents = self._inheritance_tree.get(class_fqn, [])
    for parent_fqn in parents:
        resolved = self._resolve_method_on_class_hierarchy(parent_fqn, method_name, visited)
        if resolved:
            return resolved

    return None

---

4. TDD Acceptance Criteria

• Test 1 (Inheritance extraction): Ingesting class Child(Parent): pass must emit an EXTENDS edge from Child to Parent.
• Test 2 (Inherited Method Resolution): Parent defines run(). Child inherits Parent but does not override run. Ingesting c = Child(); c.run() must successfully resolve the target to Parent.run!
• Test 3 (Decorator Extraction): Ingesting a function decorated with @mcp.tool() must record "mcp.tool" in the function node's metadata["decorators"] and emit a CALLS edge to raw_call:mcp.tool.
• Test 4 (Abstract Marker): Ingesting class Base(ABC): must set metadata["is_abstract"] = True on the class node.