Junction element with provenance node triggers crash

[juergbi]

With BuildStream 2.7.0 (#2099), a junction element with a provenance node such as

kind: junction

sources:
- kind: local
  path: .
  provenance:
    originator: bar

results in the following crash for a simple bst show:

08:41:23 [--:--:--][        ][    main:core activity                 ] START   Loading elements
08:41:23 [00:00:00][        ][    main:core activity                 ] SUCCESS Loading elements
08:41:23 [--:--:--][        ][    main:core activity                 ] START   Resolving elements
08:41:23 [--:--:--][        ][    main:core activity                 ] BUG     'NoneType' object has no attribute 'keys'

    Traceback (most recent call last):
      File "buildstream/.venv/bin/bst", line 10, in <module>
        sys.exit(cli())
                 ~~~^^
      File "buildstream/.venv/lib/python3.14/site-packages/click/core.py", line 1485, in __call__
        return self.main(*args, **kwargs)
               ~~~~~~~~~^^^^^^^^^^^^^^^^^
      File "buildstream/src/buildstream/_frontend/cli.py", line 282, in override_main
        original_main(self, args=args, prog_name=prog_name, complete_var=None, standalone_mode=standalone_mode, **extra)
        ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "buildstream/.venv/lib/python3.14/site-packages/click/core.py", line 1406, in main
        rv = self.invoke(ctx)
      File "buildstream/.venv/lib/python3.14/site-packages/click/core.py", line 1873, in invoke
        return _process_result(sub_ctx.command.invoke(sub_ctx))
                               ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^
      File "buildstream/.venv/lib/python3.14/site-packages/click/core.py", line 1269, in invoke
        return ctx.invoke(self.callback, **ctx.params)
               ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "buildstream/.venv/lib/python3.14/site-packages/click/core.py", line 824, in invoke
        return callback(*args, **kwargs)
      File "buildstream/.venv/lib/python3.14/site-packages/click/decorators.py", line 46, in new_func
        return f(get_current_context().obj, *args, **kwargs)
      File "buildstream/src/buildstream/_frontend/cli.py", line 661, in show
        dependencies = app.stream.load_selection(
            elements, selection=deps, except_targets=except_, need_state=need_state
        )
      File "buildstream/src/buildstream/_stream.py", line 168, in load_selection
        target_objects = self._load(
            targets,
        ...<9 lines>...
            need_state=need_state,
        )
      File "buildstream/src/buildstream/_stream.py", line 1684, in _load
        elements, except_elements, artifacts = self._load_elements_from_targets(
                                               ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
            targets, except_targets, rewritable=False, valid_artifact_names=load_artifacts
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        )
        ^
      File "buildstream/src/buildstream/_stream.py", line 1497, in _load_elements_from_targets
        elements, except_elements = self._load_elements([element_names, except_targets])
                                    ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "buildstream/src/buildstream/_stream.py", line 1447, in _load_elements
        elements = self._project.load_elements(targets)
      File "buildstream/src/buildstream/_project.py", line 520, in load_elements
        elements = [Element._new_from_load_element(load_element, task) for load_element in load_elements]
                    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
      File "buildstream/src/buildstream/element.py", line 1075, in _new_from_load_element
        element.__load_sources(load_element)
        ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^
      File "buildstream/src/buildstream/element.py", line 2642, in __load_sources
        provenance_node.validate_keys(project.source_provenance_attributes.keys())
                                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    AttributeError: 'NoneType' object has no attribute 'keys'

As far as I can tell, the issue is that project.source_provenance_attributes is set only as part of the second project loading pass while junction elements may already be loaded before the second project loading pass.

[juergbi]

Moving source-provenance-attributes handling to the first project loading pass should theoretically be possible, however, that would make it impossible to use include files for source-provenance-attributes, and I think we don't want that restriction.

A possible solution may be to defer provenance node validation to after element loading, but I'm open to other ideas.

[joshua-zivkovic]

would make it impossible to use include files for source-provenance-attributes, and I think we don't want that restriction

Agreed.

If source-provenance-attributes can only be extracted from the second pass config then there is no choice but to move provenance node validation later.