Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] Failing to boot (BSOD) after installing MMU3 on MK4 #3947

Open
pars3cproton opened this issue Apr 23, 2024 · 56 comments
Open

[BUG] Failing to boot (BSOD) after installing MMU3 on MK4 #3947

pars3cproton opened this issue Apr 23, 2024 · 56 comments
Labels
bug Something isn't working. electronics Issues associated with boards, board components, motors, wiring, sensors, etc. hardware issue Hardware related issue.

Comments

@pars3cproton
Copy link

Printer type - MK4 + MMU3

Printer firmware version - 6.0 RC1, RC2, Final release

Optional upgrades - MMU3

Describe the bug
After installing the MMU3, everytime the printer is turned on it tries to boot, it fails once, tries to boot again and then shows a BSOD message (the same message no matter if it's 6.0 RC1/RC2/Final release.

The only possible fix is "touching the cables" of the MMU3 while it boots, as shown in the video attached (you can also see the message in the video).
https://drive.google.com/file/d/10S5PZIgaxnuu6wRmS4JdfCftR7sWLvyO/view?usp=drive_link

With support we thought it was one of the two mmu3 boards fault, I tried replacing them but got the same error.
Then I bought another printer to mmu3 cable thinking that was the problem, but it's still showing the same behavior.

Support already has a dump file, but feel free to ask for a new one.

As of now the printer is usable if I boot it touching the cable behind the mmu3.

I tried everything, clearing MMU3 eeprom, factory resetting the printer but nothing seems to work.
@pars3cproton pars3cproton added the bug Something isn't working. label Apr 23, 2024
@m-cas
Copy link

m-cas commented Apr 24, 2024

my money is on bad solder (or cracked printed circuit trace) on the board you are pressing and the rest of the MMU unit. By pressing you are bending the material enough to restore the contact ... looks like a production defect, I'd ask for a complete new unit.

@pars3cproton
Copy link
Author

my money is on bad solder (or cracked printed circuit trace) on the board you are pressing and the rest of the MMU unit. By pressing you are bending the material enough to restore the contact ... looks like a production defect, I'd ask for a complete new unit.

I already got a replacement for both the sheep board and the pd board (big one and small one of the mmu3), replaced and doing the same. I don't need to push it hard, I just need to lightly touch it. I also bought a new mmu3 to printer cable, at this point all the electronics are replaced but still getting the same behavior.

I don't know much about boards or circuits or firmwares, but my money is on the actual printer buddy board (I have version 27)

@m-cas
Copy link

m-cas commented Apr 24, 2024 via email

@pars3cproton
Copy link
Author

this is clearly mechanical in nature. Something you are bending with your fingers enable the power to reach the MMU (leds on). You could try to push in different places (eg don’t touch the connectors, just the board; push something else, etc etc) to try to find out where is it making a difference
First of all thanks for your answers! :)
I tried touching just the cable getting inside the pd board and it boots with that too, pushing the cable inside the connector - not touching the board. If I reboot tho it doesn't boot anymore until I keep the cable pressed inside again. It's weird tho because the cable clicks and it looks in place (I don't feel any jiggle keeping it pushed or not) - also the cable is new.
I don't really know what to do at this point

@m-cas
Copy link

m-cas commented Apr 24, 2024 via email

@pars3cproton
Copy link
Author

pars3cproton commented Apr 24, 2024 via email

@m-cas
Copy link

m-cas commented Apr 24, 2024 via email

@ClaGre
Copy link

ClaGre commented Apr 24, 2024

Since touching sometimes it works and sometimes it doesn't, couldn't it be a “simple” cold solder that doesn't make contact properly?

@m-cas
Copy link

m-cas commented Apr 24, 2024 via email

@pars3cproton
Copy link
Author

pars3cproton commented Apr 24, 2024 via email

@m-cas
Copy link

m-cas commented Apr 24, 2024 via email

@pars3cproton
Copy link
Author

pars3cproton commented Apr 24, 2024 via email

@m-cas
Copy link

m-cas commented Apr 24, 2024 via email

@m-cas
Copy link

m-cas commented Apr 26, 2024

were you able to solve the issue?

@pars3cproton
Copy link
Author

pars3cproton commented Apr 26, 2024 via email

@danopernis danopernis added hardware issue Hardware related issue. electronics Issues associated with boards, board components, motors, wiring, sensors, etc. labels Apr 30, 2024
@HaWiWe
Copy link

HaWiWe commented May 1, 2024

I have the same BSOD problem when the MMU3 is connected. The fix for me is to leave the USB cable plugged into the MMU (it does not have to be connected to a computer or anything, which logically shouldn't do anything). Video of it can be found here: https://youtu.be/CG1UNtzrMik
Support is also stumped as to what the reason is.

@pars3cproton
Copy link
Author

pars3cproton commented May 1, 2024 via email

@HaWiWe
Copy link

HaWiWe commented May 1, 2024 •
edited
Loading

Glad to be of help. :) I agree with above poster who said it might be something with soldering, conductive whatever, but noone knows what exactly. As long as that works (and it does, I printed a 510 colour-change Benchy in 8:10 hours) I am okay with it. :P

@m-cas
Copy link

m-cas commented May 1, 2024 via email

@pars3cproton
Copy link
Author

pars3cproton commented May 1, 2024 via email

@pars3cproton
Copy link
Author

pars3cproton commented May 1, 2024 via email

@HaWiWe
Copy link

HaWiWe commented May 1, 2024

My friend said that that was the reason that I can't be an influencer: I also find the bug to be really funny and interesting. He said to become an influencer I would have had to make a 2-hour rant about it, namedropping other companies that do it a lot better. :P

@m-cas
Copy link

m-cas commented May 1, 2024 via email

@pars3cproton
Copy link
Author

pars3cproton commented May 1, 2024 via email

@m-cas
Copy link

m-cas commented May 1, 2024 via email

@F1rst-Lay3r
Copy link

I have a similar issue with 6.0.0 and a MMU3 on the MK4. Difference is that I think mine is not hardware related. When I revert to 6.0.0 RC2, the BSOD is gone. Re flashing to 6.0.0 and the BSOD is back. Sometimes I'm able to boot without a BSOD.

Also with 6.0.0 there are some random beeps now and then, that are also gone when I revert to 6.0.0 RC2.

For now I keep running on 6.0.0 RC2, without issues.

@thadogg
Copy link

thadogg commented May 7, 2024

Hello,
Exact same problem here. The USB cable workaround worked perfectly. This is definitely a critical bug/defect...

@F1rst-Lay3r
Copy link

Ever since I reverted back to firmware 6.0.0 RC2, I was able to boot fine without any BSOD. Up untill today that is, today I could hardly boot at all. So I tried the 'unconnected' USB cable workaround and for now it's working.

@fnordsh
Copy link

fnordsh commented May 27, 2024 •
edited
Loading

I seem to have the same or a similar issue - for me a reliable workaround is to press and hold the MMU3's reset button until the bootup of the MK4 is finished. After that, everything works fine.

image

PS: I will test the "USB cable method" when my current print is finished.
Edit: The USB cable method seems to work 9 out of 10 times.

@fnordsh
Copy link

fnordsh commented May 27, 2024

Just a thought: Is there maybe a floating input somewhere that can, depending on the read value, delay or otherwise influence the MMU3 boot process?

@m-cas
Copy link

m-cas commented May 27, 2024 via email

@sascape
Copy link

sascape commented May 28, 2024 •
edited
Loading

I've had this problem since assembly over a month ago. I thought I had assembled the MMU incorrectly but I noticed that when I hold the MMU at a different angle in my hand during boot up, I have no problems. Once it is booted up, the printer and MMU work perfectly!

As soon as my current print finishes, I will try the cable trick. It would be very funny if this fix worked.

@pars3cproton
Copy link
Author

pars3cproton commented May 28, 2024 via email

@sascape
Copy link

sascape commented May 28, 2024

Well, the cable seems to be in the right position right now (weirdly, I had the boot bug just a few hours ago). The printer boots up correctly every time I cut the power or reset.
I don't want to push my luck and I will wait for the moment the bug reoccurs.
Right now I need reliability. I'm happy to test around with the cables in 1-2 weeks when I have finished my time-sensitive projects.

@tlachmann
Copy link

Hi,
having the same issue as descrided here, https://forum.prusa3d.com/forum/original-prusa-i3-mmu3-hardware-firmware-and-software-help/pd-board-issue-overcurrent-overvoltage/

After some more investigation, I assume that this is caused by the inrush current while the capacitors are charging.
The LDO circut on the PD-board consumes also an inrush current at the same time.
I´m not sure if it is possible to softstart the MMU 24 Volts Rail or to changing the time bias of the overcurrent detection.
I assume that allowing the higher inrush current for additional 50µSec could help.

Greets Thorsten

@tlachmann
Copy link

tlachmann commented Jun 4, 2024 •
edited
Loading

yesterday I did some measurements.. my opinion, that issue is annoying...

But first: having a micro-USB cable plugged into the MMU, does not have quick-fix the behavior on my side, it still shows MMU Overcurrent

Now the annoying part, the measurements...

  • On my side, the Overcurrent issue is persistant, so if i want to boot the MK4 with mmu in factory default setup, no chance, always the Overcurrent notice.

  • Once I prepared for the current measurement, cut the three GND wires from MK4 to MMU approx 5 cm to the PD-Board and put a 2mOhms Measurement shunt in beetween which has the effect, that the Issue is gone, no longer a overcurrent failure, of course , the shunt has only 0.002 Ohms...

My assumption was then, that combining the three GND wires coming from MK4 to connect the single shunt has "fixed" the isssue.
But I was also wrong, after bridging the 2mOhms shunt, the issue was back again, persistantly, removing the bridge, results that the issue was gone...
So for me it seems, that even the 0,002 Ohms resistor limits the inrush current on the 24Volts sufficiently...

But I did also some measurements of the inrush current in the working environment.

I measured a peak current up to 25A at the moment when the 24V is switched on.
the time period above 10A is approx 1.8 to 2mSec. Then the current reduces quickly below 0.5A

See attachd screeners.

PD-Board-current - 1
PD-Board-current - 2

@m-cas
Copy link

m-cas commented Jun 4, 2024 via email

@tlachmann
Copy link

I don't know which influence the USB cable from scientific side has.
in the schematic of PCB version 0.3 the ID pin of the microUSB cable isn't connected, normaly the ID Pin controls switching from USB Slave to OTG. If it would be connected to MCU and maybe the cable has that pin grounded, then I could imagine.
For me the USB cable is more or less a esotherik thing.
image

But even the fact, that the difference of an 0.002 Ohms Resistor (some cables has higher resistance) fixes the behavior on my side, is also not explainable, at least not at that tiny value.

On the other hand, I´m sure that a resistor between 1 and 1.4 Ohms on the 24 Volts Input of the PD-Board could act as a current limiter. On the other side, this value shuldn´t also not have a influence to the Stepper functionality.

Greets Thorsten

@tschaerni
Copy link

tschaerni commented Jun 4, 2024 •
edited
Loading

I don't know which influence the USB cable from scientific side has. in the schematic of PCB version 0.3 the ID pin of the microUSB cable isn't connected, normaly the ID Pin controls switching from USB Slave to OTG. If it would be connected to MCU and maybe the cable has that pin grounded, then I could imagine. For me the USB cable is more or less a esotherik thing.

Maybe it adds a tiny amount of capacitance (which every cable has), or it just applies some pressure to the board, which would cause the same as some people have experienced (including myself) with putting small amount of pressure to the board.

Kinda a shame that the MMU3 seems to work really well, but is plagued by such a weird hardware bug.

IIRC, adding a capacitance could limit the inrush current, so a small value choke might be a better solution than a shunt resistor.

@HaWiWe
Copy link

HaWiWe commented Jun 4, 2024

For me the USB cable is more or less a esotherik thing.

I tried for a loooot of boots. So no, it is definitely not an esoteric thing but something that does work and does fix it for most of the people who have that bug even though there is no logical explanation for it. It don't find the bug too bad, rather curious. :D

@pars3cproton
Copy link
Author

pars3cproton commented Jun 4, 2024 via email

@fnordsh
Copy link

fnordsh commented Jun 4, 2024

Hi, having the same issue as descrided here, https://forum.prusa3d.com/forum/original-prusa-i3-mmu3-hardware-firmware-and-software-help/pd-board-issue-overcurrent-overvoltage/

Are you sure this is the same issue? I never got the overcurrent error, but I do get the BSOD on boot when I don't have the USB cable connected to the MMU3. Holding the MMU's reset button also helps.

"Esoteric" is a nice way to put it. ;-) In my experience, weird semi-deterministic bugs like this are often related to floating inputs. I'm sorry that I don't have much time to investigate further right now, but I would first check if all MCU input pins are initialized properly, with pullups except where they are explicitly not needed.

Next thing to try would be adding a few seconds of delay in the MMU's boot process. (Based on the observation that holding the MMU's reset until the MK4 has finished booting works reliably for me.) Not sure if I would classify that as fix or rather as a workaround, though. ;-)

@HaWiWe
Copy link

HaWiWe commented Jun 4, 2024

Yes, for every new MMU update I have a short conversation with the support, telling them it's still there and him confirming that they are still looking for the reason.

@tlachmann
Copy link

If the open USB cable would add capacitiy, then it also need to be charged on boot, which would worse the issue.

electrically that usb cable represents a antenna and maybe it acts like that and disturbes the MCU on boot which adds a delay.

@tschaerni
Copy link

If the open USB cable would add capacitiy, then it also need to be charged on boot, which would worse the issue.

electrically that usb cable represents a antenna and maybe it acts like that and disturbes the MCU on boot which adds a delay.

A crap, I meant inductance, not capacitance, that would lower the in rush current.

Yes, for every new MMU update I have a short conversation with the support, telling them it's still there and him confirming that they are still looking for the reason.

I'm also gonna get a replacement board in a few days, it'll probably be unresolved, but they asked me to send my board back so they can investigate it.

@tlachmann
Copy link

Hi, having the same issue as descrided here, https://forum.prusa3d.com/forum/original-prusa-i3-mmu3-hardware-firmware-and-software-help/pd-board-issue-overcurrent-overvoltage/

Are you sure this is the same issue? I never got the overcurrent error, but I do get the BSOD on boot when I don't have the USB cable connected to the MMU3. Holding the MMU's reset button also helps.

"Esoteric" is a nice way to put it. ;-) In my experience, weird semi-deterministic bugs like this are often related to floating inputs. I'm sorry that I don't have much time to investigate further right now, but I would first check if all MCU input pins are initialized properly, with pullups except where they are explicitly not needed.

Next thing to try would be adding a few seconds of delay in the MMU's boot process. (Based on the observation that holding the MMU's reset until the MK4 has finished booting works reliably for me.) Not sure if I would classify that as fix or rather as a workaround, though. ;-)

regardless of the message BSOD/Overcurrent notice, the symptoms looks identical, so my assumption is that both has the same cause.

@Subtle3D
Copy link

Subtle3D commented Jun 5, 2024

Had the MMU3 overcurrent issue happen on 2 MK4s.

What led to it:
It worked fine on during install and firmware updates. MMU prints would work but on starting a print it kept saying there was filament detected and needed to unload before being able to assign tools. It would ask for filament type, heat up and unload (nothing) then go to the filament selection page. Up to this point the filament sensor reading always reported off.

In the settings menu, disabling the filament sensor also disabled the MMU. Reenabling the sensor, then reenabling the MMU caused an instant reboot into MMU overcurrent.

Multiple reboots, unplugging replugging MMU, changing MMU sheep board, changing PD board, booting with each plug for steppers and Pinda sensor unplugged one at a time, caused no change. Booting without the MMU plugged in allowed the MK4 to boot, then plugging in the MMU and enabling it in setting again caused reboot to MMU overcurrent.

"Fix"
The USB cable plugged in "trick" stated above, allowed everything to boot normally and the MMU works for both of the printers (so weird).
Don't know how or if the filament sensor issue plays into this but the filament issue that led to this no longer exists on these printers either.

If the current spike on boot is non-damaging, then an interim fix until a proper hardware (if that's the issue) fix might be a firmware change to disable or increase the overcurrent detection limit for the first few seconds during MMU booting.

@Subtle3D
Copy link

Subtle3D commented Jun 5, 2024 •
edited
Loading

Addition to my previous post.

I have several other MK4's with MMU3's that also had the filament detection issue on MMU3 install and disabling/enabling the filament sensor and MMU in the settings page stopped the detection issue and did Not cause an MMU overcurrent. (Definition of insanity? Same actions different results :)

@tlachmann
Copy link

Today I had a chance to measure current when the MMU overcurrent error appers, and compared to my previous non-error measurements, I can´t measure an overcurrent.
in my previous measurement an peak overcurrent upt to 25A (for a very little time) happens, now the printer detected an overcurrent, while only 10,10A was reached (yellow line 20,20mV/2mOhms Shunt = 10,10A) , and switched off the 24 V Supply when it reached only 12-13V (blue line)

I think now it seems it is a wrong positive interpretation in the software.

99F5CE39-D8CE-480D-ABBC-CB6BED99799E_1_102_o

@tlachmann
Copy link

tlachmann commented Jun 5, 2024 •
edited
Loading

I spend some time to take a brief view into the code, and found that normally the MMU is "soft" powered up with a pulsed power-up procedure.
But I didn´t see anything that looks like a pulsed power-up procedure.. So it seems that the issue is really software based and the pulsing didn´t happen, or it happens but the electronic isn´t able to execute, what I think is more realisitc.

Because in the code the voltage will be switched on for 5µSec and off for 70µSec, this repeats for 15000µSec.
But the problem is, the voltage needs under none fault condition approx 1000µSec to rise from 0 to only 5 Volts, 3500µ secounds from 0-24Volts.... that wouldn`t work.

Source:

Prusa-Firmware-Buddy/src/mmu2/mmu2_power.cpp

Line 20 in b1b77f6

// The code below pulse-charges the MMU capacitors, as the current inrush 

using namespace buddy::hw;

// The code below pulse-charges the MMU capacitors, as the current inrush
// would due to an inferior HW design cause overcurrent on the xBuddy board.
// In case overcurrent would still be triggered, increase the us_total
// value to pre-charge longer.
static constexpr uint32_t us_high = 5;
static constexpr uint32_t us_low = 70;
static constexpr uint32_t us_total = 15000;

  if (!config.can_power_up_mmu_without_pulses()) {
      CriticalSection critical_section;

      for (uint32_t i = 0; i < us_total; i += (us_high + us_low)) {
          {
              buddy::DisableInterrupts disable_interrupts;
              MMUEnable.write(Pin::State::high);
              delay_us_precise<us_high>();
              MMUEnable.write(Pin::State::low);
          }
          delay_us(us_low);
      }
  }

  MMUEnable.write(Pin::State::high);

@tlachmann
Copy link

I hope Prusa also recognize that behavior. For today released 6.0.2. FW the changelog doesn`t show anything.

I also investigate regading the pulsed prodcedure, and after changing to a larger shunt, I could observe some artifacts that lokks like the pulses, before it was hidden by disturbances.
But of course the artifacts are so minor, that it doesn`t take account to reduce inrush current.

you can see the artifacts by the pulsed procedure in the small slope of the blue line before it rises vertical

89582564-60D8-465A-A5F9-61629682F61C_1_102_o
41D74A9E-A6CD-4375-BD62-03904F34515E_1_102_o

@tschaerni
Copy link

I few days ago I received a new xBuddy board and a new MMU3 board (and a new power panic cable, as I have an earlier revision with the 4 screw gearbox cover).

After replacing them, it seems to work without issue so far (crossing fingers).

Maybe the Issue is mitigated with newer versions of the xBuddy board?

Could be helpful to gather information about the revisions of the printers/xBuddy boards?

@GvnCampbell
Copy link

I contacted support as well with the same issue. We did a bunch of tests. If I unplugged the cable from the mmu3 it still happened. Once I fully removed the cable from mmu3 and buddy board it would boot up fine. So they think it was the cable and shipped me a new one that comes tomorrow.

I did share them links to this issue though in case. We will see if the new cable works.

@tschaerni
Copy link

I contacted support as well with the same issue. We did a bunch of tests. If I unplugged the cable from the mmu3 it still happened. Once I fully removed the cable from mmu3 and buddy board it would boot up fine. So they think it was the cable and shipped me a new one that comes tomorrow.

I did share them links to this issue though in case. We will see if the new cable works.

Hm... I didn't had that issue, it worked fine with the cable connected to the xBuddy board but not the MMU3, it even worked fine with the PD Board connected to the cable, but disconnected from the MMU3 board.

But yeah, maybe you just have a faulty cable, even tho I can't really imagine the cable being the culprit, unless there is a short in the cable somewhere.

@GvnCampbell
Copy link

I contacted support as well with the same issue. We did a bunch of tests. If I unplugged the cable from the mmu3 it still happened. Once I fully removed the cable from mmu3 and buddy board it would boot up fine. So they think it was the cable and shipped me a new one that comes tomorrow.
I did share them links to this issue though in case. We will see if the new cable works.

Hm... I didn't had that issue, it worked fine with the cable connected to the xBuddy board but not the MMU3, it even worked fine with the PD Board connected to the cable, but disconnected from the MMU3 board.

But yeah, maybe you just have a faulty cable, even tho I can't really imagine the cable being the culprit, unless there is a short in the cable somewhere.

Well I can confirm that its not the cable. I got the new cable. Replaced it and same issue.

I can keep hitting the reset button and it eventually boots up but its still very annoying.

Back to support I go.

@m-cas
Copy link

m-cas commented Jun 13, 2024 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working. electronics Issues associated with boards, board components, motors, wiring, sensors, etc. hardware issue Hardware related issue.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests