Feature: Configuration - support wildcards and regex for array definitions

[gupichon]

Description, motivation and use case

The current configuration requires explicitly listing all element IDs in array or device definitions. This becomes fragile and hard to maintain when new elements are added or naming conventions evolve.

Two complementary improvements are proposed:

1. Pattern-based selection inside elements lists (wildcards, regular expressions and exclusions).
2. Python-based configuration macros, allowing scripted generation of configuration blocks directly from YAML.

This issue focuses on extending the configuration layer to support:

• Wildcards and regular expressions inside elements.
• A simple negation syntax for exclusions.
• A scripting mechanism (elements_code) allowing dynamic generation of configuration entries using embedded Python.

The goal is to improve scalability, maintainability and expressiveness of the configuration layer while preserving full backward compatibility.

---

Part 1 — Pattern-based element selection

Proposed solution

Keep the existing elements key unchanged and make the element string itself expressive.

Supported syntax:

Syntax	Meaning
pattern	inclusion using exact match or wildcard
~pattern	exclusion using exact match or wildcard
re:<regex>	inclusion using regular expression
~re:<regex>	exclusion using regular expression

Examples:

elements:
  - BPM_C*-*
  - ~BPM_C04-*
  - re:^BPM_C1[0-9]-[0-9]{2}$
  - ~re:^BPM_C10-.*

---

Interpretation

Given the following example:

elements:
  - BPM_C*-*
  - ~BPM_C04-*
  - re:^BPM_C1[0-9]-[0-9]{2}$
  - ~re:^BPM_C10-.*

The evaluation proceeds as follows:

1. Include all BPM_C*-*
2. Remove those from cell C04
3. Add elements matching the regex
4. Remove elements matching the exclusion regex
5. Sort the resulting list according to the reference ordering defined in the accelerator devices section

This ensures that array definitions remain stable and consistent with the canonical accelerator device ordering.

---

Resolution rules

• elements remains a list of strings.
• Each entry is resolved independently.

Rule grammar:

rule := ["~"] ("re:" <regex> | <glob>)

Resolution semantics:

• Rules are evaluated sequentially.
• Positive rules add matching elements.
• Negative rules remove matching elements.

Matching operations always use the reference ordering defined in the devices section of the accelerator configuration.

The resulting list:

• preserves the reference ordering of matched elements
• removes duplicates deterministically

Example:

elements:
  - BPM_C*
  - ~BPM_C04*

This allows selective reinsertion of elements while keeping the accelerator reference ordering.

---

Matching rules

Pattern interpretation:

• re: prefix → regular expression
• string containing * or ? → wildcard (glob)
• otherwise → exact element ID

Resolution must:

• Use the device order defined in the accelerator devices section as the canonical ordering.
• Be deterministic.
• Clearly define behavior for 0 matches (error or empty contribution — must be specified and consistent).
• Remove duplicates deterministically.

Full backward compatibility shall be preserved.

---

Part 2 — Python-based configuration macros

Motivation

While wildcards and regex improve selection, some configurations require:

• Nested loops
• Multiple parameterized ranges
• Systematic naming patterns
• Complex device definitions
• Computed attribute values

For such cases, pattern matching is insufficient. We therefore propose a controlled macro mechanism using embedded Python code.

This was discussed as an additional proposal.

---

Proposed solution: elements_code

Introduce a new optional key:

elements_code: |
    <python code>

The embedded Python code:

• Must return either:

  • A dict (single configuration entry), or
  • A list[dict] (multiple entries).

• If a list is returned, it is expanded into the surrounding list.

• The macro block is replaced by the returned structure before normal parsing continues.

This mechanism follows the same philosophy as existing file-expansion logic: detection → execution → replacement → continue parsing.

---

Example — BPM generation

devices:
  - elements_code: |
      out: list[dict] = []
      ranges = [(4,33), (1,4)]
      for r in ranges:
          for cell in range(r[0], r[1]):
              for elem in range(1, 11):
                  bpm = {
                      "type": "pyaml.bpm.bpm",
                      "name": f"BPM_C{cell:02d}-{elem:02d}",
                      "model": {
                          "type": "pyaml.bpm.bpm_simple_model",
                          "x_pos": {
                              "type": "tango.pyaml.attribute_read_only",
                              "attribute": f"srdiag/bpm/c{cell:02d}-{elem:02d}/SA_HPosition",
                              "unit": "m",
                          },
                          "y_pos": {
                              "type": "tango.pyaml.attribute_read_only",
                              "attribute": f"srdiag/bpm/c{cell:02d}-{elem:02d}/SA_VPosition",
                              "unit": "m",
                          },
                      },
                  }
                  out.append(bpm)
      return out

The resulting list is injected into the configuration tree and parsed normally.

---

Execution semantics

The macro mechanism must:

• Execute before object construction.

• Run in a controlled execution environment.

• Provide:

  • A minimal safe namespace.
  • Optional helper utilities (e.g. range, math, etc.).

• Enforce that the return type is either dict or list[dict].

• Raise a clear configuration error otherwise.

The parsing process becomes:

1. Load YAML.
2. Detect elements_code.
3. Execute code.
4. Replace macro block with returned structure.
5. Continue parsing recursively.

---

Determinism and reproducibility

To preserve reproducibility:

• Execution must be deterministic.
• No implicit access to external state unless explicitly allowed.
• No hidden side effects.
• Ordering of generated elements must be preserved as returned.

---

Backward compatibility

• Existing YAML files remain valid.
• elements_code is optional.
• No change to current schema validation for standard entries.
• Pattern-based elements resolution remains independent.

---

Considered alternatives

Separate external Python generation script

Instead of embedding Python inside YAML, users could generate YAML files using standalone Python scripts.

Pros:

• Clear separation of code and configuration.

Cons:

• Breaks self-contained configuration principle.
• Harder to track provenance.
• More complex CI workflows.

The embedded macro approach keeps configuration declarative while allowing structured generation when necessary.

This approach will always remain possible, independently of the macro mechanism proposed here. Nothing in this proposal prevents users from generating YAML files externally using Python (or any other language) before loading them into pyAML.

---

Dedicated DSL instead of Python

A domain-specific language could replace Python.

Pros:

• Safer, constrained syntax.

Cons:

• Reinvents control structures.
• Higher implementation complexity.
• Lower expressiveness.

Given Python is already the core language of the project, leveraging it is consistent with project philosophy.

---

Checklist

• I've assigned this issue to a project
• I've @-mentioned relevant people

[gupichon]

Ok, you can look at EBSOrbit.yaml and sr.yaml for usage examples. I’ve discovered that the element order can be an issue. I wasn’t able to find a proper way to describe magnet arrays using regex.
Issue #199 will have a significant impact on these configuration options.

[gupichon]

@JeanLucPons, I would like to add macros to describe elements.
The idea would be to use a detection mechanism similar to the one we already have for file expansion. The macro would contain some Python code that returns either a valid dict or a list of dicts. The result would replace the macro block. If it returns a list, it would be expanded into the surrounding list.
This was mentioned during the discussions, I believe by @simoneliuzzo.

For the BPMs, it would look like this:

devices:
  - elements_code: |
      out:list[dict] = []
      ranges = [(4,33), (1,4)]
      for r in ranges:
          for cell in range(r[0], r[1]):
              for elem in range(1, 11):
                  bpm = {"type": "pyaml.bpm.bpm",
                      "name": f"BPM_C{cell:02d}-{elem:02d}",
                      "model": {"type": "pyaml.bpm.bpm_simple_model",
                                "x_pos": { "type": "tango.pyaml.attribute_read_only",
                                           "attribute": f"srdiag/bpm/c{cell:02d}-{elem:02d}/SA_HPosition",
                                           "unit": "m",
                                           },
                                "y_pos": {"type": "tango.pyaml.attribute_read_only",
                                          "attribute": f"srdiag/bpm/c{cell:02d}-{elem:02d}/SA_VPosition",
                                          "unit": "m",
                                          },
                                }
                      }
                  out.append(bpm)
      return out

[gupichon]

Ok, I’ve validated this behavior. Now, any key or list element can be replaced by a code block, as in my example. Would you prefer that I open a separate issue, or should I just update the current description?
@JeanLucPons
@simoneliuzzo
@TeresiaOlsson

[gubaidulinvadim]

@gupichon, why do we need this in the config file? I would prefer to have some utility function devices.record('config_filename.yaml') that can save/add/remove devices to an existing config file.

devices:
  - elements_code: |
      out:list[dict] = []
      ranges = [(4,33), (1,4)]
      for r in ranges:
          for cell in range(r[0], r[1]):
              for elem in range(1, 11):
                  bpm = {"type": "pyaml.bpm.bpm",
                      "name": f"BPM_C{cell:02d}-{elem:02d}",
                      "model": {"type": "pyaml.bpm.bpm_simple_model",
                                "x_pos": { "type": "tango.pyaml.attribute_read_only",
                                           "attribute": f"srdiag/bpm/c{cell:02d}-{elem:02d}/SA_HPosition",
                                           "unit": "m",
                                           },
                                "y_pos": {"type": "tango.pyaml.attribute_read_only",
                                          "attribute": f"srdiag/bpm/c{cell:02d}-{elem:02d}/SA_VPosition",
                                          "unit": "m",
                                          },
                                }
                      }
                  out.append(bpm)
      return out

Edit: Could you give an example where element order is an issue? The elements can be sorted with respect to the order in the lattice file (s-position) to resolve some (most?) ordering issues.

[gupichon]

The goal is to keep all devices in a shorter and more readable file. With a utility function like devices.record('config_filename.yaml'), you still need to build the devices explicitly. More importantly, this approach is complementary to regex, because the rule required to build an array in the correct order cannot always be expressed as a regex. I realized this when I tried to build the EBSOrbit.yaml magnet arrays using an expression.

With this proposal, it would be somewhat like having macros in the configuration file, but with more flexibility. It could be used everywhere (arrays, device lists, and so on).

A utility function like the one you suggest could be complementary.

[gubaidulinvadim]

I'm not against this feature. It could be useful, but this is not a very readable config to most end users. That being said, a gigantic list of devices is also not very readable. I'm also not sure if your macros solution is compatible with database-compatible config files that some people want.

The order issue in your particular case can be solved by sorting the elements in an array in order of position/index in the lattice file. But this will require a mandatory lattice file in the config and may force all element arrays to always be sorted by index in the lattice.

[simoneliuzzo]

The order issue in your particular case can be solved by sorting the elements in an array in order of position/index in the lattice file. But this will require a mandatory lattice file in the config and may force all element arrays to always be sorted by index in the lattice.

I think that we would not do any harm making the assumption of lattice order. It is something hardware defined (written in stone). Why choosing any other order? In case some other elements are added a new configuration file is prepared with the new lattice order.

[gupichon]

For a database-compatible configuration, we will need to develop an entirely new feature anyway.

Also, none of this is mandatory. For example, at SOLEIL, we will continue generating the configuration file and hardly ever open it. This approach is mainly useful when you want a small configuration file for a specific use case.

In my opinion, the main use cases will be:

• Scripted configuration building
• Generated configuration files
• A new configuration plugin linked to a database

By the way, with the catalog feature I’ve started developing, I will be able to implement a Tango plugin that does not require specifying any control system devices in the configuration—only PyAML objects (CFM, BPM, etc.). You will be able to refer to them by their Tango names, and the plugin will retrieve all the necessary information directly from the Tango database.

I think that we would not do any harm making the assumption of lattice order. It is something hardware defined (written in stone). Why choosing any other order? In case some other elements are added a new configuration file is prepared with the new lattice order.

I agree, except that some institutes do not have a lattice file at all. You have to make sure it works without a lattice file. The problem I would like to solve is how to have the simplest possible configuration file containing all the necessary information. You can achieve the same result by running your own build script (which is what I personally prefer). Ultimately, it depends on how you want to manage it (YAML-only configuration, Python scripts, or a mix of both).

[gubaidulinvadim]

I know it's optional :)

I agree, except that some institutes do not have a lattice file at all. You have to make sure it works without a lattice file. The > problem I would like to solve is how to have the simplest possible configuration file containing all the necessary information.

We shouldn't believe them :) @TeresiaOlsson, you do have a lattice file, right? There's one in BESSY II examples. From what I understand, it's just not representative of the machine design anymore, but the order of the elements is correct. Anyhow, the order of elements in the lattice can be checked by going inside the tunnel and looking at them.

For me, the simplest solution seems to have regex/wildcards in the config file and sort the elements when the array is created. But maybe there are some situations (I can't think of any now) where sorting is not by the index. If there are other reasons than sorting, then "macros" are ok.

[TeresiaOlsson]

I think this feature will be very useful. It will lower the threshold for new users that need to make a configuration from scratch a lot.

I always thought the database interface could be implemented as simple as just extracting a json and then that is read into pyAML in the same way as any other json? I think inside the database you want the arrays to be extended into the full element names, but the users who don't have a TANGO database and need to create a config database from scratch need to populate it somehow. For that I think this feature would also be useful. Then you could just write a minimal initial configuration, read that in and have something to start building your database from.

I think keeping the lattice file optional is very important. There are many use cases for using pyAML as a pure hardware abstraction layer without a simulator.

For me if everyone anyway need to write their own script to generate the yaml file because it's so long and tedious to do, what is then the point of the yaml file? Then it's starting to feel to me like an obsolete middle step and it would have been better to keep the MML way of doing the configuration where it is programmed?

[TeresiaOlsson]

I know it's optional :)

I agree, except that some institutes do not have a lattice file at all. You have to make sure it works without a lattice file. The > problem I would like to solve is how to have the simplest possible configuration file containing all the necessary information.

We shouldn't believe them :) @TeresiaOlsson, you do have a lattice file, right? There's one in BESSY II examples. From what I understand, it's just not representative of the machine design anymore, but the order of the elements is correct. Anyhow, the order of elements in the lattice can be checked by going inside the tunnel and looking at them.

For me, the simplest solution seems to have regex/wildcards in the config file and sort the elements when the array is created. But maybe there are some situations (I can't think of any now) where sorting is not by the index. If there are other reasons than sorting, then "macros" are ok.

No, the BESSY II lattice file is good and represents the lattice as it is now. It's more for machines like linacs where there will be no pyAT lattice file. If we put having a lattice file as a requirement that will exclude all applications for linacs until we have implemented interfaces to other simulators.

[gupichon]

For me if everyone anyway need to write their own script to generate the yaml file because it's so long and tedious to do, what is then the point of the yaml file? Then it's starting to feel to me like an obsolete middle step and it would have been better to keep the MML way of doing the configuration where it is programmed?

I believe these approaches are complementary. We should also clearly distinguish the various use cases (production, development, CI/CD, etc.), since they may require different strategies. Additionally, as @TeresiaOlsson mentioned, this should simplify configuration file creation for new users. At least, that is the intention.