Create some sane external behaviour of the scripts

[pointhi]

@jkriege2, @hackscribble, @SchrodingersGat, @poeschlr

We are a small group of people which created most of the script generated parts inside KiCad. First of, thanks for using my stuff ^^.

In contrast to the framwork itself, I'm merging contributions into scripts with almost no review. Simply because most scripts are used mostly a few times by mostly the same people.

This results in some collection of code, where it's hard to use/change existing scripts, as well as the invention of scripts which get better placed into the framework itself (like the KLC definitions).

I would like to start a discussion about how much we can unify our development of scripts, to help new contributors as well as ourself.

The main proposal could be described as separate code and data. This is a main statement when talking about secure coding, but has other reasons as well. Everyone invented his system to get the data into the system. (I invented at least 3 different ones ^^). This means people can not simply use those data for other purposes (like generating 3d-models), and people are required to decrypt the way of data storage to push their own values into the system.

I added ModArgparser.py as my proposed fix for this problem into the framwork. Although no one really uses it right now beside me. This class would allow us to unify the external behaviour of all the scripts, means everyone can push data to it using .csv or .yml files. Furthermore, every script get's the same comandline interface, for interaction.

I would ask about your opinion about those problems, as well as what are you thinking about my proposal. I'm especially thinking about the usage of ModArgparser for 3d-file generation as well. This would allow us to use the same part definitions for both tasks.

[poeschlr]

Seems like a good idea. Currently i am a bit short on time to look into it in great detail. If i don't report back in the next few weeks give me another notification. (I might forget about this otherwise.)

[ghost]

Hi @pointhi

I'm also short of time at present, but here are some initial thoughts - probably raising more questions than giving answers 😞

1. Your framework is a tremendous contribution to the KiCad ecosystem. Many thanks for creating and supporting it.

2. I agree with the idea of separating code from data. I would also separate the core framework from scripts that use it. I suggest that your current repo should just have the framework, documentation on how to use it and example script(s). The real scripts would be better in a separate repo, maybe kicad-library or a new repo kicad-library-sources (like packages3D-sources).

3. For info, these are the factors that have influenced how the design of my scripts has developed over time:

• Python factory model (because I adopted this for my 3D model scripts, where it allows a clean separation between creating the FreeCAD models and exporting them in the two formats)
• YAML configuration file shared between footprint and 3D model scripts (of course there is a lot of data in the file not used by the footprint script)
• hierarchical YAML file (so that common parameters for a group of similar components only appear once, e.g. https://github.com/pointhi/kicad-footprint-generator/pull/36/files#diff-43d77d46c671ed8a00935800c3370580)

4. We need a better way of recording how footprints were created: script or manual, and which script (and which version). At the moment, finding this out relies on the collective memory of the librarians or by trawling back through commits and comments on PRs. It would be great if the KiCad footprint file structure allowed for metadata to be included, but I don't think it does.

5. Question about policy. If we end up with a standard set of script designs (covering program structure, how parameter data is provided, how to execute them), do we make their use mandatory for scripted footprints? Or would we still accept footprints scripted a different way?

[poeschlr]

After playing with my latest contributions i would suggest the following:
Have data and code separated as you suggest here.

One yaml file responsible for configuration stuff (KLC settings)

• should hold the format strings for lib and footprint naming
• should hold stuff about silk, fab, courtyard line thicknesses, offsets, ...
• maybe also make it possible to configure the font sizes for reference and value fields. (maybe even enable something like my latest scripts where the user can add an unlimited amount of reference and value copies on any layer just by adding stuff to a config file.)

Another yaml file then holds the dimensions of the parts to be produced. (I wrote it such that there could be multiple of these files.)

This should allow us to easily change the generator scripts to conform with newer versions of the KLC
It should also allow users to generate footprints that conform with their personal preferences.
And it should make it easy to add additional packages.

Edit: the config file (KLC) should be select-able via an optional command line option. I chose --config (-c). I think this should be standardized.

[grob6000]

Keen on the idea of sharing input between footprints and 3d; a 1:1 alignment is the policy at the moment anyway.

[s-light]

is there any news on this?
i am currently on the way to understand and find all thinks and scripts i have to know of to add some parts...

where is the best place for this to get started?
i initially started at kicad-pcb.org - contribute
but it was not clear enough to me that it would make sense to use the linked scripts (and how to use the scripts) to just add one QFN part ;-)
@poeschlr kindly explained this to me

so if iam understanding this correct - currently i have to add the parameters for the 3d package also to the script at https://github.com/easyw/kicad-3d-models-in-freecad/blob/master/cadquery/FCAD_script_generator/QFN_packages/cq_parameters.py
and run the script to generate the (missing) packages... ?
if this is the wrong place to ask pleas excuse.
is it better if i (re)post this to forum.kicad.info ?

[poeschlr]

Yea the contribution page could use a bit of rework. (but we at least mention the scripts there so it is not nothing ;) )

If you want to add the 3d model then the place you list is the correct one.

This might not be the best place to ask regarding something to do with the policy of the official lib or with the 3d models. (A better place would be one of our repos, or even the forum as i am quite easily reachable that way.)

[s-light]

@poeschlr you mean the official library repos?
next time i will first ask in the forum... :-)
thanks for answering here.

[pointhi]

I created some initial proposal: kicad-python/wiki/new-Footprint-wizard-Plugin-System

very rough, many things to think about. It should incorporate things like config_KLCv3.0.yaml but in a more flexible way. For example, support name overwrites where people can specify how a resistor name should be built upon given values (IPC naming convention).

Beside footprints, we also need to think about how to handle symbols and 3d-models in the future. I would like to build a system where we can generate a whole custom library using cmake in the end. And the same script can also be used in the "footprint wizard" as well.

[Misca1234]

Is .yaml something we should use with the 3D models as well?
LED_SMD is using it, but the rest is using cq_parameter.py style.
Perhaps we should convert cq_paramter.py to .yaml ?

[poeschlr]

A direct conversion is not always possible. (At least not easily)
Most cq_parameter files have calculations defined in them so there the separation is not so good. (This means it is a bit more work than simply copying parameters from one format to the other)

[matthijskooijman]

Just looked through this discussion and the proposal by @pointhi, all looks a good direction. However, it seems there is not much movement in this direction at the moment. I also wonder what the best approach is here.

Right now, it seems the plan is to:

1. Improve separation between code and data
2. Improve separation between footprint-generating code and boilerplate (loading and interpreting yaml files, parsing commandline, writing output files).
3. Use kicad-python bindings to let KiCad generate footprints, rather than generating text-files directly.
4. Integrate as a wizard into KiCad, so (some) footprints can be generated on-demand.

However, it seems an attempt is made to solve all four of these at the same time, which makes it quite a complex thing, which might be causing the lack of movement.

Would it make sense to leave 3. and 4. for later and focus on 1. and 2. instead (keeping 3. and 4. in mind, of course)? So just keep using the current KicadModTree for output and the commandline for calling scripts, but define better helpers to reduce script boilerplate?

Doing so would probably simplify the problem, but solving 1. and 2. would allow any effort spent on (new) scripts right now to be more future-proof, and a lot easier to adopt once 3. and 4. are also implemented.

It would not even be needed to convert all existing scripts right away (just a few to validate the approach, probably), but having a single "blessed" style for new scripts would already greatly help new contributors.