Problem: After firmware upgrade one motor does not power up and rates are off #485
Comments
|
Note that Y steps_per_mm says 200, but that is because I was playing around with some settings and forgot to reset before $CD, but that is NOT what is saved in the file. I can snag the file with $localfs if you need me to. |
|
Update: Reverted firmware and then tried the next several releases.
|
|
That controller is unique with daisy chained TMC5160 drivers and the SPI use_enable. It is not something I could reproduce here. The supplier said they were sending us a least one several months ago, but we never got one. If the resolution is off, it is likely the microstepping value is not getting to the motor driver. I can try some basic TMC5160 checks next week. |
|
Anything I can do to help debug? I could spin up the dev tools if needed. |
|
|
|
Try v3.5.0-pre5. It has a number of fixes to problems that resulted from the switchover to the new framework. That switchover happened in v3.4.5 so your problem could be related. |
|
I built main a couple days ago and that didn't help but I'll try pre5 as well and report back, |
|
The only reason to build from main would be if you needed to change something in the compile-only configuration, such as adding OLED support. Every time that we update main, we do an official release. main is never ahead, and usually behind, our work in progress. |
|
pre5 is slightly better. Looks like the rate and distance is correct again but one of the motors is still not operating. |
|
Might have been wrong about Pre5 being slightly better. Is there anything I can do to help debug? Do you suspect this is a framework bug? I'm way out of my comfort zone with regard to hardware but I think all of the drivers are daisy-chained yet only the two that are ganged together are having a problem. I was suspicious that cs_pin had to be set for both motors but the firmware specifically told me I was wrong. If you can point me in the general direction I should be able to put in some debugging code and collect some data. |
|
Debugging by remote control is too slow and too subject to misinterpretation. If you can find some way to get an XPro into our hands we can solve the problem. Otherwise it will always be on the back burner because the process is too inefficient to displace all of the other things we are also doing. |
|
Fair answer. I understand that it's not practical if it's back and forth copy-paste. I'd be more than happy to roll up my sleeves and give it a go but I'd need some bootstrapping in the form of "this is how it's supposed to work". 40+ years as a software engineer, coded c++ professionally for 15 of them however quite rusty these days. I messaged the XPro people but that didn't prove to be effective. I can't justify the expense but if there are enough other XPro people out there, I'd be happy to contribute. It would be more cost effective for me to buy a 6-pack and had I known about FluidNC before I started, that's what I would have done. The next best thing I can do is get you ssh access to my setup but I'm fairly sure I know what you'll think about that idea. Either way, thanks for FluidNC. It fantastic! |
|
Okay this is how it is supposed to work. I really don't have time to write this but... We use the third-party TMCStepper library to access the TMC devices. It has a complicated class structure that tries to capitalize on the similarities between different TMC device families while also handling their differences. That is a tall order because, in my opinion, the Trinamic hardware engineers did not understand how to design their register interfaces to accommodate the needs of software. TMCStepper is pulled into the PlatformIO compilation setup via a "lib_deps" line in platformio.ini The TMCStepper library sits on top of the Arduino framework, which is a set of C++ classes that tries to make it possible to use Arduino coding principles and associated Arduino libraries with ESP32. That framework includes an esp32-hal - layer that interfaces between the Arduino C++ classes and the C-style device drivers provided by Espressif's ESP-IDF SDK. ESP-IDF has C device drivers that are themselves built on top of yet another HAL layer that tries to abstract away differences between the classic ESP32 chip and new variants like ESP32-S2 and even the version that is a RISC-V instruction set instead of Xtensa. And that ESP-IDF HAL layer does not go directly to the hardware; instead it invokes yet another "ll" (low level) HAL-ish layer that is a bunch of always-inline functions with a layer of #define "functions" on top. The ESP-IDF source code can be inspected if you know how to find the actual version that is included in the Arduino framework package. That inclusion is in the form of precompiled ".a" library files, not as directly-compilable source. It is possible to build ESP-IDF from source but that requires a different platformio.ini setup, which adds yet another layer of complexity to the compilation environment and risks getting versions that do not reflect the code you are trying to debug. With the default precompiled setup, you cannot make code changes at the ESP-IDF level for debugging purposes. The Arduino framework stuff that TMCStepper uses is primarily the SPI class, used for, obviously, sending commands over the SPI bus. The Arduino framework's SPI class driver uses some driver stuff from ESP-IDF but also does some stuff by itself. The SPI bus is shared between the SD card and the TMC devices. That requires using a different chip select (CS) GPIO for SD vs TMC. That gets complicated because TMCStepper wants to use the Arduino digitalWrite() function to control the TMC CS GPIO, whereas for the SD Card, we use the FluidNC "Pin" framework that lets us use a uniform way to access different "abstract GPIOs" including ones that implemented via I2S. To make the TMC CS work, we overload a weak function in the TMC Stepper code and redirect it back through the FluidNC Pin framework via a "PinMapper" class. The old version of ESP-IDF does not have the internal HAL layer because it only supported classic ESP32 chips. The new version has the internal HAL, and also they changed the behavior of some functions at the driver layer. The new Arduino framework's esp32-hal- layer has changes that reflect that changed behavior at the ESP-IDF driver layer. There are potential bugs at both places. There are also potential problems due the use of a newer version of GCC and the associated C++ standard libraries, especially in the area of atomics. The problem that you are experiencing could be caused by changes or bugs in any of those layers. Also, we do not entirely understand what the XPro people have done with their hardware. Now do you see why I am less than thrilled by the prospect of spending a lot of time being your guide through the wilderness? |
|
Well that's a mouthful and more than I expected, so thank you for that. You've outlined the complexity well and it looks quite daunting. Abstractions on top of abstractions on top of a binary-only library doesn't bode well. |
|
Not to mention the fact that there are at least 6 major components designed by entirely different people with different architectural styles, goals, and approaches, and you end up needing to comprehend both old and new versions of many of them in order to navigate the change set. |
|
As I'm sure you predicted, I didn't get very far. That being said, I opened an issue on the SparkConcepts GitHub about this and it might have resulted in them sending an XPro to each of you. They say at least one of the units was already successfully delivered. |
|
I received the one on Friday. There are a few design issues that make it not 100% compatible with FluidNC at this time. Some of those should be able to be overlooked to allow basic motor and USB functionality, but I still have issues with both. I am optimistic we can modify FluidNC to make it work, but it could take some time. To get the flexibility we need could force major rewrites of the SPI and Trinamic sections. Please be patient and remember that FluidNC is free and open source firmware. We are are not affiliated with Spark Concepts. This is the first time we have seen this hardware or the schematics. |
|
@bdring Well understood and thank you! |
|
I received the xpro yesterday and an looking into the SPI issues surrounding daisy-chained SPI for the driver chips. |
Controller Board
CNC xPro V5
Help From Board Vendor
Machine Description
Bulkman 3D Ultimate Bee, configured for dual X motors.
Configuration file
Startup Messages
User Interface Software
UGS
What happened?
I was a few revisions behind and did the upgrade to 3.4.7.
After doing so, one of my X axis motors does not power up. I can turn the stepper by hand, while the others I am not able to. After $MD, I can turn them all.
The feed rates and distances are also not correct. G91 G0 Y100, for example, moves about 1/4 of the expected distance and not at the expected rate. steps_per_mm looks right, but I wasn't smart enough to make a backup of the config before the upgrade so unless the upgrade process changed something, it should be right.
I haven't tried to revert the firmware (yet).
Other Information
No response
The text was updated successfully, but these errors were encountered: