Various comments & repports

[thomaslepoix]

Hi! I have few little things to report from a first try, so I just open one issue.

---

I took a look at field_dump.py and I saw the dumps always use the Et_ prefix, disregarding the dump type, while it should be reserved for electric field in time domain.

---

I find the names used for PML materials, appearing in AppCSXCAD, ambigous. Here names are like PML_i, i being an index identifying each material. In the OpenEMS terminology, the same form is used PML_n, n being the number of cells compounding the PML. So I suggest you to use something like PML_n_i instead.

---

There is no margin between the structure edges and the boundaries, so when using a PML boundary condition, a part of the structure is in the PML, that is not meant to happen according to the OpenEMS documentation. (This problem does not exist with MUR and PEC conditions.)

---

Otherwise I'd like to say I think you go in the right direction to ease the use of OpenEMS and to create what will become its "next generation interface". The automatic mesher you are working on is actually the worst lack of OpenEMS, an optimizer would also be very nice.

I develop Qucs-RFlayout that currently export OpenEMS Octave scripts and I am considering using Python/pyems in future.

Plus, I want to provide a kind of generic script (through a script generator library?), relevant for most of the simulations (I focus on PCB things), so I started working on a script CLI and on a set of basic postprocessing functions (plot S parameters, plot feed point impedance, plot far field radiation, etc.). In fact I would like to provide some basics to somewhat2openems converters developers, to in the end harmonize scripts interfaces from the user side. I think this idea and your project could converge.

Finally, with some other community members, we started the Open-RFlab project to improve the free & open source RF design ecosystem. If you are interested in joining the discussion circle, you are totally welcome. :)

[matthuszagh]

Thanks for the feedback! I'll address these in the same format as the post.

---

I agree with this. Et_ is misleading for non Efield time dumps. This is a simple fix.

---

I see why this is confusing, and I agree with your solution. This is also a trivial fix. It's probably worth mentioning a little about PML here. One of the reasons I omitted this originally is that I only ever use PML_8, so it was unambiguous to me. According to Thorsten, this is the only PML you ever should use, but this information is buried in the forum. As a separate issue, pyems should document this since the OpenEMS wiki is in a read-only state (see my post here about adding this info to the wiki) and will be replaced entirely at some unknown future time.

---

This PML behavior is intentional and correct. However, I believe it's worth documenting this in pyems, because the OpenEMS wiki is (in my opinion) ambiguous on this point. The PML here is used to create a perfect match at the ports so there are no signal reflections. This has 2 benefits: the simulation will finish more quickly because the energy will decay faster (i.e. once it leaves the region inside the PML) and it will lead to more accurate simulation results since S-parameters, impedance, etc. won't measure signal reflections. Now to more directly address your concern: the reason we can't place it outside the structure is that then it will create a reflectionless boundary for the surrounding air layer, not the microstrip. This will present an open circuit at the port ends and the simulation will only terminate when enough energy has radiated away into the air or dissipated as heat due to the lossiness of the trace and PCB (i.e. it'll take a very long time). If you want to experiment with this, you can do so by passing something larger than 8 to the expand_bounds parameter. Thorsten does the same thing in microstrip notch filter tutorial. It's worth noting that an alternative would be to move the PML away from the structure and terminate the ports with an impedance equal to the specified characteristic impedance. However, this does not work nearly as well as the PML (I've tried) and therefore suffers greatly on the 2 points mentioned above. Thorsten explicitly mentioned this use of the PML in the forum here. Again, this is critical information that should be in the wiki.

It's also worth mentioning that while structures can and should be placed in the PML boundary, field excitations, probes, lumped elements, etc. should not. This is already taken care of by pyems, which will trigger an error if this is the case. Additionally, the structure inside the PML should not change along the length of the PML. Pyems should identify when this is adhered to and when it isn't, though it doesn't. I will look into adding this feature. A related feature which I might add is an "intelligent" extension of the structure from an existing structure to adhere to this condition in cases where the structure would otherwise violate it.

A final point on this topic is that I dislike that the PML appears by default when AppCSXCAD is opened. I'd rather that it be invisible at first, with the ability to turn it on (in the way that ports are presented, for instance). I believe this requires an upstream patch to AppCSXCAD/QCSXCAD though, which I'm not sure when I'll get to.

---

I love the idea of better microwave tooling that is also interoperable. I've joined your discord and look forward to seeing how this progresses. Please feel free to use pyems as an extension to Qucs-RFlayout. It sounds like that functionality could be quite useful. One warning though: since pyems is still in its early stages, the API will likely continue to change for some time (I don't expect to change the major architecture, but function/class signatures are likely to change). If you find that some part of your tool breaks because of a pyems API change, feel free to post an issue and I'll work with you to find a solution.

As a final point (since you mentioned optimization) I think it's worth discussing what pyems's goals are in this regard. minimize (which just delegates to scipy) is already quite functional. I've used it in a number of cases with good results. This uses local minimization (not global), which I believe is adequate since structures should start pretty close to their optimal dimensions. Something that would be nice, though I haven't looked into it yet, would be to make use of parallel simulation runs in the optimization, leveraging pyems's ability to run multiple simulations completely in parallel (see sweep). This will require some thought since this would probably require ditching scipy, and might require writing a custom optimization algorithm.

In regard to optimization of the simulation (not structure) I'm also working on another function to compute the optimal value and simulation convergence of a mesh parameterization. For instance, to determine a good choice for the mesh min_lines parameter and what the value would have been if a large number had been used here. This would, in theory, allow you to determine a highly accurate simulation result without the very long simulation that would result from using a very small mesh resolution. This idea is meant as a formalization of Thorsten's advice to keep adding mesh lines until the simulation results don't change. In any event, this functionality is not yet ready, though some of the preliminary functionality is implemented.

[thomaslepoix]

OpenEMS is so subtle..

Ok! So you are way more knowledgeable than me about boundary conditions, that's alright!

Do you have an idea of how AppCSXCAD/QCSXCAD handles the default visible state? Using Octave, the boundary conditions are not shown at all. Taking a look to a CSX file produced by a pyems script and an other produced by a Qucs-RFlayout generated Octave script, both looks quite different. (I link it here if it could help you lpf.xml)

Octave functions add this paragraph :

  <FDTD NumberOfTimesteps="300000" endCriteria="1e-05" f_max="5000000000">
    <Excitation Type="0" f0="2510000000" fc="2490000000">
    </Excitation>
    <BoundaryCond xmin="PML_8" xmax="PML_8" ymin="PML_8" ymax="PML_8" zmin="PML_8" zmax="PML_8">
    </BoundaryCond>
  </FDTD>

While Python ones add a material with particular properties for each PML :

        <Material ID="4" Name="PML_1" Isotropy="1">
            <FillColor R="211" G="211" B="211" a="200" />
            <EdgeColor R="211" G="211" B="211" a="200" />
            <Primitives>
                <Box Priority="-1">
                    <P1 X="5.000000e+00" Y="-2.500000e+00" Z="-5.693310e-01" />
                    <P2 X="3.400000e+00" Y="2.500000e+00" Z="1.792546e+00" />
                </Box>
            </Primitives>
            <Property Epsilon="1.000000e+00,1.000000e+00,1.000000e+00" Mue="1.000000e+00,1.000000e+00,1.000000e+00" Kappa="0.000000e+00,0.000000e+00,0.000000e+00" Sigma="0.000000e+00,0.000000e+00,0.000000e+00" Density="0.000000e+00" />
            <Weight Epsilon="1.000000e+00,1.000000e+00,1.000000e+00" Mue="1.000000e+00,1.000000e+00,1.000000e+00" Kappa="1.000000e+00,1.000000e+00,1.000000e+00" Sigma="1.000000e+00,1.000000e+00,1.000000e+00" Density="1.000000e+00" />
        </Material>

Athough I would prefer the boundary conditions behave like the dump boxes rather than being simply absent from AppCSXCAD..

---

I will watch evolution of the structure optimizer and the mesh optimizer. I am particulary interested in the mesher.

I'm glad you joined! Thank you for your advice but I have things to end before supporting pyems so I don't worry about your API stability.

[matthuszagh]

Oh, I see the confusion. The PML boxes appear because I add them explicitly (relevant code here). I use them as visual aides - they have no simulation effect. They're implemented as an air layer with a negative priority. I added this in during development because I was fed up with counting mesh lines to see where my PML's ended and if some structure that shouldn't be in the PML ended up there.

Needless to say, AppCSXCAD does not display PML cells. Ideally AppCSXCAD would do this instead of pyems since it's a bit of a hack in pyems. And I think seeing the PML is a useful feature. But, implementing this in AppCSXCAD will be more work than the pyems fix. Morevoer, pyems needs to do this work anyway to signal errors to the user if they place a structure in the PML.

As for why it's displayed by default, I think AppCSXCAD displays all material primitives and turns of non-material primitives by default. Though this is a guess, I haven't really looked into it. If this is the case, allowing certain structures to not display by default will probably involve extending the csx file format in addition to otherwise modifying the source code. I'm not sure if Thorsten would want that.. and in that case it would probably be best to implement this properly by getting appcsxcad to display the pml.

[thomaslepoix]

Oh ok, I didn't see that your csx file only contains the <ContinuousStructure> section! But why doesn't it also contain the <openEMS> section at top level and also the <FDTD> one?

[matthuszagh]

Good question I have no idea. I'd never noticed that, although it has been a while since I've used the octave interface. I wonder what (if any) functionality the FDTD section provides, since I was under the impression the csx file was really just about the physical structure. Maybe openems can run a simulation from this file on its own because it includes all the simulation directives? I'd be curious to know if thorsten's python examples exhibit this behavior too, or I've done something to somehow cause it to be omitted.

[thomaslepoix]

Yes, that's it. Without this section, AppCSXCAD can read it but OpenEMS no. The OpenEMS CLI expect it and the Octave API use the CLI. So I guess the C++/Python API bypass this mechanism.

I think it is a mess to use a mix of an API communication between the script and the softwares and of a file based one (required by AppCSXCAD I guess) since it leads to an uncomplete simulation csx file, keeping some informations in the script.

I am wondering if the OpenEMS API provides a prefered way to append the <FDTD> section to the file.

[matthuszagh]

Based on a very cursory investigation, it appears that all of the work done to write <FDTD> is performed in octave. Therefore, to perform the same thing in python would require reimplementing the same code as opposed to just extending the Cython interface to support an existing C++ function. For reference, here's the C++ function to write an xml file. The two relevant octave files appear to be struct_2_xml and octave_struct2xml_2.

I agree with you on aesthetic grounds and I wonder why Thorsten chose to do it this way. Out of curiosity though, do you have a practical reason for wanting to have the FDTD data embedded in the file? For instance, maybe you're hoping to invoke the CLI directly on the file?

---

Separately, I've implemented both suggested changes (commits here and here) and created issues for some of the other possible improvements discussed above.

[thomaslepoix]

Nice!

I am just thinking of interoperability. For example I simulated with OpenEMS the same Qucs microstrip schematic using Qucs-RFlayout and QucsStudio (that is closed source, written in C++, and totally wraps OpenEMS without many customization possibilities). Both produced different results, so I had to work at AppCSXCAD/OpenEMS level to investigate and understand what caused such a difference, but it was possible.

In fact I think each layer (script generator, script, xml files) should be independent from others. I might need it later but my mind is not clear yet about it. Nevermind :)

[matthuszagh]

That's a really interesting example. I'd be curious to experiment more with how similar setups generate different results. And I wasn't aware openems was the RF backend to QucsStudio, so thanks for pointing that out. As a side note, that same issue was one of the motivations for creating pyems and trying to enforce correct setup. I noticed as I was performing simulations that I could get results that seemed reasonable, but were sufficiently wrong because I'd missed something subtle about the simulation setup (mostly in relation to mesh generation).

Yep, I'm inclined to believe you're right in this regard. I imagine interoperability will only increase if the stages work independently of one another. In any event, if/when it is needed I don't this should be too hard to add.

[matthuszagh]

A related thought to yours occurred to me about the benefit of a fully independent xml simulation file. It would make it easier for myself and others using pyems to recruit help from thorsten and others in the forum. We can't really provide a pyems file as a MWE, but the xml file could be used here. This obviates the need to attempt to recreate what pyems does using the native python interface, which will become increasingly difficult as pyems becomes more sophisticated. The other option I'd considered previously would be to have pyems log the openems and csxcad invocations it uses so that this could be used and tweaked as a MWE. But, this would be barely readable and I like the standalone XML solution much better.

[thomaslepoix]

QucsStudio uses the regular Qucs backend but since the 3.x release (July 2020), it can also use OpenEMS on microstrip components, that looks like this.
It was precisely the same schematic trough different converters producing different setups, almost only the microstrip shapes were the same. So of course the result were different. The question being : Why is this setup better than this one? What should be a good setup?

Yes, to use OpenEMS, one has to learn a lot and dig for knowledge about its subtilities. I think both of us want to avoid that.
People build high level layers to avoid OpenEMS trickyness. interoperability and compatibility between those is a good requirement not to create other oddities, plus it can enlight some, like this one, already present in OpenEMS. It is an occasion to fix them if they become problematic.

I think I got it but what does MWE stand for?
You are totally right. I am facing the same problem with Qucs-RFlayout, altough produced script only use the regular Octave interface, with all the sugar a script easily up to 1500 lines of codes with language tricks, etc.. Not so readable neither in the end.

And I'm just thinking on a postprocessing utility that could also be based on a csx file reading (like AppCSXCAD and openEMS), and so eventually adding a <PostProcessing> section to its format to store the few missing informations. But that's a whole new topic.

In fact, looking at this CSX format, I am turning myself to think it is good and working on it / improving AppCSXCAD, OpenEMS and the APIs is probably better than building new layers above the whole and working around its oddities. For example you are coding a mesher while AppCSXCAD has a button "Detect edges" that does not apply the thirds rule. It's like a first draft yet never finished of where Thorsten would have liked to place a mesher, and of course we would have liked to find it.
Of course this thought is pertinent only for few features and not every ones we want to add, but may we should consider to implement back some features where they should be, when we will be satisfied and have the time to do it.

[matthuszagh]

Sorry for the confusion, MWE is minimal working example. The idea being, if we find openems bugs through pyems/qucs-rflayout/etc. we could provide the complete xml file rather than having to do a lot of work to turn it into a minimal python/octave file using the official interface.

Yeah the thought of implementing this work upstream had occurred to me and it was one of the reasons for keeping the license compatible. I think there are a few benefits to keeping it separate at least at the beginning though. Speaking for pyems, it gives me more room to experiment, especially with ideas that other people may not want in their initial form. Additionally, I can consider the interface on its own merits, rather than trying to conform to what people already expect. Also, Thorsten has expressed in the past that he's not sure an effective automesher is really possible. So, I think he would be more willing to accept one if it had gone through a lot of testing. This is not to say this should never be merged back into openems, but i think at least they should start separate. In regard to stuff like we were discussing with the xml format, i think this is definitely a candidate for implementing directly in the upstream repos.

[matthuszagh]

I'm closing this since I've created individual issues for the various things mentioned here. But feel free to reopen if I've missed something or if you've got more feedback.