v1.0 testing

[gkasprow]

The boards start with an overheating reset chip. There is a bug in pin assignment.VSOT package drawn in datasheet has nothing to do with SOT-23. Use MSOP package pinout instead... It is not critical for board operations so we can remove it for now.

[gkasprow]

POE supply works as expected, at least without load

[gkasprow]

My switch supports 802.3at or 802.3af. I connected 5.1R resistor at the output of the PoE module and it does not work. With 8.R load resistor, it works fine. Idle current from 12V supply is 60mA.
The 802.3at delivers up to 25.5W, 12V@5.1R gives already 28V so this could not work.
12@8.1R is 17.7W so it is within the limits of the standard.

[gkasprow]

For some reason, the PoE does not work with dummy injectors. In such case, one must buy also dummy PoE splitter and use DC jack power entry instead. The PoE converter is too smart and does not accept such power source.

[jordens]

That's good. Those passive injectors are also 12 v only typically.

[hartytp]

@gkasprow can we open an issue for each change, rather than one long "errata" issue please? I find that with long issues, it's easy for some of the changes to get missed.

[hartytp]

Also, mixing test reports with change requests makes the problems a little worse. Let's keep testing to a separate issue (ideally, one issue for each test).

[hartytp]

Thinking about PWM resolution. Here is some feedback from @gkasprow:

We are limited by lowpass filter cutoff frequency and PWM counter
length. The filters are dual pole ones with Fc=159Hz.
It also means that if we want to have an attenuation of the filter of
let's say 90dB, we should work at 30kHz.
The PWM counter can be even 32-bit wide but this results in very low
pulse frequency. In TM4C the PWM modulator can be clocked up to 80MHz.
To have 30kHz output we have to use a counter that counts to 2666. It
essentially means an 11bit resolution.
If we replace the capacitors with 1uF, then 90dB attenuation will be
achievable with 3kHz PWM pulses, which means the counter can count to
26,666 and we get almost 15bit resolution.

I'd like the keep the poles roughly where they are now, because I don't want them to impact the loop latency, which is currently set by the 5Hz ADC sample rate. If we want the PWM filter cut-off to be at least 10 times the ADC sample rate the it must be at least 50Hz, which is not much below the current value.

I'll do a simulation with 11 bits resolution. I suspect we will find the PWM resolution is a significant limitation in the overall thermal stability we can achieve (particularly for well-insulated masses where we want to run the TEC driver at relatively low currents).

I can see two ways of getting around this. (1) be a bit cleverer with the PWM to squeeze out another 3-4 bits of resolution (e.g. update it more than once per ADC cylce) (2) add small 16-bit I2C DACs like AD5693R to drive the TEC controller (these can easily be retrofitted to existing boards).

@gkasprow @sbourdeauducq preferences?

[gkasprow]

DAC is usually a better approach because it's not trivial to get very high attenuation from the RC loop filter.

[jordens]

From my experience the relevant cutoff determining the stability will not so much be the reconstruction filter/PWM resolution alone nor the ADC but the PI corner and the overall P gain. And as mentioned in the other issue, there are a bunch of bits to be gained in noise shaping, i.e. not just plain PWM or the interpolating PWM you mention, Also, what's the difference to your existing solution that is achieving 5 ppm stability (18 bit) without issue with the same PWM method?

[hartytp]

Also, what's the difference to your existing solution that is achieving 5 ppm stability (18 bit) without issue with the same PWM method?

Currently, the short-term stability (seconds) is typically quite a bit worse than the longer-term stability. For one setup with a heater current of approx 100mA, we see quantization noise in the from of oscillations of around 400uK pk-pk. The longer term stability (hours) is much better than that. Adding a bypass resistor in parallel with to increase the heater reduces the issue significantly.

[gkasprow]

PWM output stability is determined mainly by the CPU power supply which is affected not only by the supply rail voltage but also CPU load (drop on bonding).

[hartytp]

In our prototype we use a 40MHz SYSCLK with a 10kHz PWM period, updating the PWM once per ADC sample. So, resolution is approx 12 bits. Based on experience (although this is strongly dependent on the heat capacity of the load etc), a few extra bits would be enough to make the TEC current resolution fine in most cases. tl;dr I'd like to shoot for 16 bit resolution in the heater current.

I don't have a strong preference between PWM + interpolation or DACs.

Also, what's the difference to your existing solution that is achieving 5 ppm stability (18 bit) without issue with the same PWM method?

I don't follow you here? The overall stability and the noise induced by the PWM are not directly related.

PWM output stability is determined mainly by the CPU power supply which is affected not only by the supply rail voltage but also CPU load (drop on bonding).

That's true, although I've never directly observed any issues due to this. Over short timescales, noise is removed by the RC filter and over longer timescales it's taken care of by the feedback.

[hartytp]

My inclination right now is that it's probably easier to add 16 bit DACs than to worry about it. Any objections to that?

I'd use a MSOP -> pin header breakout board to hack the DACs onto existing Thermostat prototypes. Thoughts?

[gkasprow]

The board space is not an issue. Such DAC needs a good reference.

[hartytp]

AD5693R has an internal reference.

[hartytp]

Here is a toy simulation of thermostat (needs checking and tidying up!): https://gist.github.com/hartytp/3aa5c0098564771335e6920a95466b6e

The simulations match our observations with prototype hardware well. In particular:

• with 11-bits, the PWM can easily be the dominant limitation. The more I think about it, the more I'm pro having a 16-bit DAC in the next revision.
• with single-precision floats the numerics can easily be the limitation. We should either use double-precision floats (the control loop is simple and slow, so this can be done in software) or need to be careful with numerics

[hartytp]

Linearising the temperature calculation around the set-point to remove the logarithm makes things a little better. But for genuine mK stability, it's probably best to stick with double precision maths unless there is a good reason not to...

[jordens]

I don't follow you here? The overall stability and the noise induced by the PWM are not directly related.

You were worried about just that: "PWM resolution is a significant limitation in the overall thermal stability". That's precisely quantization noise.
I'm not so worried about getting another 4 bits out of the PWM but adding the DAC is also fine by me.
If you add the DAC, make sure you can PWM its output with the CPU. It's a simple change and gives multiplicative resolution (i.e. just what's needed for large gain dynamic range) while also supporting both approaches.

[hartytp]

You were worried about just that: "PWM resolution is a significant limitation in the overall thermal stability". That's precisely quantization noise.

I think we were talking past each other. Let's forget about it.

If you add the DAC, make sure you can PWM its output with the CPU. It's a simple change and gives multiplicative resolution (i.e. just what's needed for large gain dynamic range) while also supporting both approaches.

Here, you essentially just mean making sure that the DAC can be updated at, say, 10s of kHz and has a proper reconstruction filter before the TEC controller, right?

[jordens]

I'd like the DAC output to be PWM-ed before it hits the reconstruction filter. That's in effect a hybrid multiplying DAC where one (DAC or PWM) sets the range and the other is the PI actuator. In either case it increases dynamic range. You can leave the PWM always on when you are not interested in that dynamic range. And you can keep the PWM frequency high if you are worried by the reconstruction filter.

[hartytp]

@jordens yep, that sounds fine to me. From a hardware perspective, it's what I was planning already. From a software perspective, I probably wouldn't bother to implement DAC PWM in the first instance -- just set it to a fixed value once per ADC reading/control loop iteration. We can play with PWM in firmware down the line when a use-case emerges that needs it.

[jordens]

Just to be clear: I am not referring to muxing the DAC control word but a mux after the DAC that switches between the DAC output and whatever zero TEC current is. Then toggle that mux FET with a CPU PWM output. Feed the mux output into the filter.

[hartytp]

What's the advantage of that over just toggling the DAC control word between two values in software? Modulation of the DAC control word at something like 10kHz should be easy to implement on the microprocessors using interrupts, shouldn't it. Wouldn't that be enough for what you want to achieve?

[jordens]

I think you are underestimating the complexity (race conditions, CPU load, timer setup, SPI/DMA) of modulating the DAC control word. And with that DAC speed at 10 kHz you can only gain 2-3 bits dynamic range.

[hartytp]

I think you are underestimating the complexity ... of modulating the DAC control word

Probably, I've not given it much thought since it's not something I expect to need (16-bit TEC current resolution should be good enough for all the use-cases I have in mind).

I'm not against adding the modulation circuit you describe to the next design revision if you think it will be genuinely useful in real-world applications. As always with these things, if you haven't already done it, I would urge you to do a simulation/get some data for the application you have in mind to make sure that the extra complexity is really worth it. No point increasing circuit complexity if it doesn't help a real use-case.

Also, I don't anticipate contributing to the hardware/firmware development costs for the PWM circuit since we won't use it.

But, with those caveats, and as long as there aren't any issues with board space etc, I don't object to adding this in the next design revision.

[hartytp]

• If we do add the DACs, we should make sure that we can use solder jumpers/population options to go back to plain old PWM.

[gkasprow]

What about adding to the DAC output the PWM signal, but attenuated below 1 LSB?

[hartytp]

I'm fine with that. It also gives us an easy way of bypsassing the DAC and reverting to PWM if we need to. @jordens?

[jordens]

Maybe I am misunderstanding, but that addition is prone to transients. They would be as large as the PWM quantization that you are worried about. IME precision adding to gain resolution is nasty here. Multiplication isn't since the range factor is fixed.

[hartytp]

Obviously, it all depends on how much time the student has...

[jordens]

Yes, send me one. I'd also like to give the ddsm-dma-pwm hybrid a try one the stm32 version once that's ready.

[gkasprow]

He has to defend the thesis pretty soon so not much time is left. I asked him to write a thesis in English so we would have good documentation.

[hartytp]

No worries. The fact that he's doing even basic testing is great for the project.

[wobucho]

I've added pt100/pt1000 option and fixed problem with Static IP connection. I've tested also DHCP option and it works well. I'll tidy up the code and upload it soon.

[hartytp]

Great! Let me know when it’s done and I’ll have a play with it!

[hartytp]

@gkasprow we'd like to do some tests on the V1.0 design this week finalising the changes for v2.0. Please could you ask your student to post his code here as well as his thesis? Once we have that, we'll have a play and post results.

cc @RHanley1

[gkasprow]

The firmware is here

[gkasprow]

and the thesis is here
Thesis_wobucho.pdf

[hartytp]

Thanks!

[wobucho]

This is firmware for the version with Thorlabs chips. I will upload firmware for MAX1968 version. However, I used TI's Tivaware libraries v. 2.1.3.156 (I replaced lwIP 1.4.1. with lwIP 2.1.2). Unfortunately, they discontinued to provide this version of Tivaware and there were some troubles with compatibility between the microcontroller and the newest version of Tivaware (I don't know if they fixed it). Just in case, I will upload also Tivaware 2.1.3.156 (with lwIP 2.1.2.) to my google drive, so you will have the access to full firmware that was tested by me.

[hartytp]

Cool, thanks. Can you also post binaries and flashing instructions please?

[wobucho]

I have uploaded the firmware in the firmware branch. Binary output file is in the Debug folder and it is called thermostat_max.bin. Tivaware library is on my google drive. If you want to work with the code using Code Composer Studio, you need to import the project to the workspace and change the value of SW_ROOT to the folder TivaWare_C_Series-2.1.3.156 downloaded from the google drive.

If you simply want to flash binary file (thermostat_max.bin), you need to use CCS Uniflash.

Before flashing any program though, you need to set the MAC address. As far as I know, it is possible only with the use of XDS debug probe. Optionally you can use EK-TM4C1294XL board.

[hartytp]

Before we send the new revision to manufacture we should do a few more tests on the current hardware.

Most critically, I would like to see a plot of the measured resistance when a 10k low-temp co resistor is connected in place of the RTD. A plot of that over a few hours tells us how good the stability of the temperature controller will be. Ideally, we should do this both with a 12V PSU connected via the barrel connector and with Thermostat powered from PoE.

@wobucho @gkasprow are you able to have a look at that? If not, we'll look at it some time during the next couple of weeks. (out of curiosity, @jordens have you had a play with it yet?)

If we have time, a few other things which would be good to check (all with a precision 10k resistor in place of the RTD):

• cross talk between the two TEC drivers and the ADC (sweep the TEC set current and look at the change in measured resistance)
• temp co of the design (use one channel to stabilise the board, while another logs the precision resistor value)
• EMI sensitivy: put the precision resistor on the end of a few m of screened twisted pair and wave some noise sources around (50Hz transformer/drill/etc)

[jordens]

I did play with it, flashed it, hooked up a rtd and a peltier and went through the interface. See the issues I filed. Did get the software to work well enough to make statements about anything else though.

[hartytp]

Didn’t get?

[jordens]

Yes.

[gkasprow]

I'm afraid my student won't do anything because he finished his thesis and has a full-time job.

[hartytp]

That’s fine. Well do it when we can (next week or two)

[hartytp]

Maybe a silly question, but why are you storing the MAC addr in the user regs? Aren't there dedicated MAC address registers one can use?

[gkasprow]

We have a dedicated MAC chip (IC5) on the Thermostat PCB

[hartytp]

@gkasprow I'm talking about v1.0, where I thought the TCM4 had an internal MAC and PHY.

I'd ideally like to do a little testing on the v1.0 hw using your student's code before we finalize the v1.1 design. If I can get the code up and running reasonably quickly then I'll test the v1.0 hardware. If I have too many issues, I won't test v1.0 and will skip straight to v1.1 for which M-Labs are writing fw...

[hartytp]

@jordens am I right in thinking that all I need to do is write the MAC addr to user reg 0 and user reg 1 and then flash the binary. After that the code should work well enough for basic testing of the ADC / TEC driver? Or did you have issues when you played with it?

[jordens]

Yes. Just set the registers. About the software you are on your own: As I said and explained in the issues I filed, I couldn't get it to work well enough. It did something that could be called regulating the temperature but the design issues of the PID make using it extremely hard/impossible. And even when it was "stable" there was a sizable deviation and fluctuations. The newer commit with the 1ms interval hangs/crashes quickly.

[hartytp]

ACK. For testing, I'm more interested in open-loop measurements, so a dodgy PID implementation isn't an issue.

FWIW my view is that I don't want to sink large amounts of time into writing FW for the v1.0 hardware. So, if I can't get Greg's student's cope doing what I want relatively quickly then I'd rather press on with v1.1 production.

[jordens]

If the thing crashes all the time then it doesn't matter whether you are looking at open loop or closed loop.
IMO quite a lot of software (loop, network, configuration, adc configuration/interface, conversion for the PWMs etc) could be developed on this version already that would be very reusable for the next version.

[wobucho]

The firmware hangs? I am aware of issues with PID design, but I have not experienced hanging since I fixed the problem with memory leak in MQTT part.