Override metadata on the command line

[prosilio]

I'm trying to override version metadata on the command line, but I haven't found a method that works. Using the regex example, let's say that I want to point input to the file "VERSION" instead of "src/mypackage/__init__.py". I added --config-settings=metadata.version.input="VERSION" to my pip install command, but it threw an error (below). What's the proper way to specify metadata on the command line? Based on the Config Reference, I think that this should be possible, right?

scikit-build-core version is 0.11.6.
Python version is 3.10.12.

+ Exception Group Traceback (most recent call last):
|   File "/home/user/my-project/.venv/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 389, in <module>
|     main()
|   File "/home/user/my-project/.venv/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 373, in main
|     json_out["return_val"] = hook(**hook_input["kwargs"])
|   File "/home/user/my-project/.venv/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 157, in get_requires_for_build_editable
|     return hook(config_settings)
|   File "/tmp/pip-build-env-iwusxd_c/overlay/lib/python3.10/site-packages/scikit_build_core/build/__init__.py", line 177, in get_requires_for_build_editable
|     requires = GetRequires.from_config_settings(config_settings)
|   File "/tmp/pip-build-env-iwusxd_c/overlay/lib/python3.10/site-packages/scikit_build_core/builder/get_requires.py", line 80, in from_config_settings
|     return cls(_load_scikit_build_settings(config_settings))
|   File "/tmp/pip-build-env-iwusxd_c/overlay/lib/python3.10/site-packages/scikit_build_core/builder/get_requires.py", line 67, in _load_scikit_build_settings
|     return SettingsReader.from_file("pyproject.toml", config_settings).settings
|   File "/tmp/pip-build-env-iwusxd_c/overlay/lib/python3.10/site-packages/scikit_build_core/settings/skbuild_read_settings.py", line 391, in from_file
|     return cls(
|   File "/tmp/pip-build-env-iwusxd_c/overlay/lib/python3.10/site-packages/scikit_build_core/settings/skbuild_read_settings.py", line 207, in __init__
|     self.settings = self.sources.convert_target(ScikitBuildSettings)
|   File "/tmp/pip-build-env-iwusxd_c/overlay/lib/python3.10/site-packages/scikit_build_core/settings/sources.py", line 632, in convert_target
|     raise ExceptionGroup(msg, errors)
| exceptiongroup.ExceptionGroup: Failed converting tool.scikit-build (1 sub-exception)
+-+---------------- 1 ----------------
  | Traceback (most recent call last):
  |   File "/tmp/pip-build-env-iwusxd_c/overlay/lib/python3.10/site-packages/scikit_build_core/settings/sources.py", line 601, in convert_target
  |     tmp = source.convert(simple, field.type)
  |   File "/tmp/pip-build-env-iwusxd_c/overlay/lib/python3.10/site-packages/scikit_build_core/settings/sources.py", line 409, in convert
  |     return {k: cls.convert(v, _get_inner_type(target)) for k, v in item.items()}
  |   File "/tmp/pip-build-env-iwusxd_c/overlay/lib/python3.10/site-packages/scikit_build_core/settings/sources.py", line 409, in <dictcomp>
  |     return {k: cls.convert(v, _get_inner_type(target)) for k, v in item.items()}
  |   File "/tmp/pip-build-env-iwusxd_c/overlay/lib/python3.10/site-packages/scikit_build_core/settings/sources.py", line 408, in convert
  |     assert not isinstance(item, (str, list, bool))
  | AssertionError
  | Field tool.scikit-build.metadata
  +------------------------------------

[LecrisUT]

What happens if you have all fields populated in the pyproject.toml and you use the flags to overwrite?

[prosilio]

Thank you for the suggestion. I tried it with and without the input field populated in pyproject.toml. Same error in both cases.

Ideally, I could add some print statements or use a debugger to figure this out on my own, but so far I haven't managed to point pip to my local build of scikit-build-core. I'll keep trying, but in the meantime I really appreciate any ideas you might have.

[prosilio]

Just after I wrote my last comment, I managed to get my local copy of scikit-build-core to work. Here's what I found...

The problem occurs in src/scikit_build_core/settings/sources.py in the ConfSource.convert() method, which is obvious from the error message, but I wasn't sure why AssertionError was being triggered. Here's what I get when I print(f"{raw_target} : {item}"):

# raw_target   : item
<class 'dict'> : {'version.input': 'VERSION'}
<class 'dict'> : VERSION

If I understand this correctly, I think that the correct sequence should actually be

# raw_target   : item
<class 'dict'> : {'version': {'input': 'VERSION'}}
<class 'dict'> : {'input': 'VERSION'}

Is there a mistake in my --config-settings=metadata.version.input="VERSION"? Or is this a bug in scikit-build-core?

[prosilio]

It looks like the root problem is that ConfSource.get_item() does not handle dotted input beyond stripping off the "metadata." prefix.

EDIT: Removed some half-baked code that I'll iterate on in a PR.

[prosilio]

I think that the scope of this problem may be beyond my very limited understanding of scikit-build-core. Here's what I've found, in case it's of use to other developers:

My initial idea about turning dotted strings into nested dictionaries doesn't work in general, as it breaks inputs such as cmake.args and cmake.define. I had the nested dict idea in the first place due to this line in src/scikit_build_core/settings/sources.py:

tmp = source.convert(simple, field.type)

Printing simple and field.type, I saw

simple: {'version.input': 'VERSION'}
field.type: typing.Dict[str, typing.Dict[str, typing.Any]]

When the ConfSource.convert method is called on these inputs, it expects a nested dictionary (based on field.type), but it gets dict[str, str].

I don't know if the best way to handle this issue would be to change field.type to typing.Dict[str, str] for metadata fields or to change all dotted fields (such as cmake.args) to something like typing.Dict[str, typing.Dict[str, typing.Any]].

[LecrisUT]

I think you are on a good track, but I am not sure exactly which parts are failing there. Could you start by extending the tests in

scikit-build-core/tests/test_settings.py

Line 265 in 10de684

def test_conf_nested():

Then you could enter the debugger and step through the logic there.

I don't know if the best way to handle this issue would be to change field.type to typing.Dict[str, str]

That should not be necessary. Afaik typing.Dict should just superclass dict. We might need to extend the handling of the typing annotation variables though.

If you think the issues is in

scikit-build-core/src/scikit_build_core/settings/sources.py

Lines 597 to 599 in 10de684

	for source in self.sources:
	if source.has_item(*prefixes, field.name, is_dict=is_dict):
	simple = source.get_item(*prefixes, field.name, is_dict=is_dict)

I would suggest to try and see what the TomlSource does in that loop and try to recover that behavior.

Hope this helps, and thank you for all the work in debugging this