[Brainstorming]Adding support for Dremel Idea Builder (i.e. FlashForge Dreamer)

[bbatchelder]

I own a Dremel Idea Builder. I'd really like to get OctoPrint to work with it. I am a software developer, but not much Python experience, and the OctoPrint codebase is entirely new to me...so I could use some guidance.

Dremel has provided me with a snippet of C++ code that demonstrates how to push a file to the SD card over USB and initiate printing.

A few things I am noticing at first glance:

• It looks like the Dremel is expecting a tilde (~) as a start character for each command.
• It looks like the Dremel is expecting a \r\n instead of just a \n
• It looks like the M28 command (load file to SD) expects a file size in addition to destination path (the M28 I see in util/comm.py is only sending a filename).
• It looks like the Dremel is expecting an "M601 S0" be sent prior to every command.

I am thinking that as a first step, I'll just pull the latest code and modify util/comm.py to do the above, and see if I get anywhere. If that goes poorly, I'll probably fall back to writing a proof of concept app in C# (the language I am most useful in).

Here is a Dropbox link to the C++ snippet I was given.

Here are some conversations I've found online that add some additional information:

Google Groups thread where some folks sniffed the commands being sent to do various things, including upgrading the firmware.

Simplify3D Forum thread where a user explains how they set up a firmware profile for the Dreamer, and mentions the requirement for M601 S0 and the tilde start delimiter. Though they mention the end delimiter being a single LF (and not the CR/LF I saw in the C++ snippet).

[bbatchelder]

I've got rudimentary communication with the printer working using pyusb/libusb. I'm now writing a plugin modeled after the GPX plugin that provides a serial factory for communication.

I'll keep plugging away at it.

[foosel]

That looks very interesting. I just had a look at the Google group thread you linked and it looks like the commands are sent without checksums and line numbers (which makes sense if the communication transport layer is reliable).

I was recently fiddling with the communication layer in OctoPrint to hunt down some bugs and noticed I was still missing a mechanism in there to disable checksum stuff altogether. Would it help to add something like that? It would be simple to add. An alternative serial implementation then would not have to strip that stuff again.

@markwal would that maybe be also interesting for the gpx plugin? I have to admit, I don't know if that uses line numbers oror not.

[markwal]

Yes, might be helpful since it'd save the time it takes to compute the checksum and for me to strip it back off again. I'd need to change some stuff around about how I figured out when printing has begun, but no problem really.

[bbatchelder]

@foosel I honestly don't know enough right now to know if that would help :-) I'm guessing if it would help @markwal it will help me.

Right now my biggest issue is having to make sure I send a "M601 S0" regularly (perhaps in front of every command) because otherwise the real commands are ignored. Which means I have to strip out the response to the M601, as OctoPrint won't know what to do with it. I have no idea why FlashForge would design it to work this way, to the point where I wonder if I am missing something.

Is there perhaps a better way to handle that (i.e. a way to get OctoPrint to want to send the M601 in front of every legit command, so that it will expect and handle the response)?

Additional issues:

• The M20 command to list SD card contents doesn't seem to actually bring anything back. I'm not sure if I am giving it the correct path, though. It has both onboard SD and a SD slot. Is there a standard for this? I've looked around but not found anything.
• When it comes time to upload a file to SD, it is expecting file size to be given as part of the M28 command. Is there a straightforward way to get that from OctoPrint?

Thanks for the help!

[foosel]

Just to make sure I understand that correctly, working communication looks something like this?

> M601
< ok
> some command
< ok
> M601
< ok
> some other command
< ok

Or rather this:

> M601 some command
< ok
> M601 some other command
< ok

M20 usually takes no parameters in "usual" printers. I'd try something like M20 /. If that works, your plugin could simply swap regular parameter less M20 with path including ones through a gcode queuing phase hook.

Same should work for the M28. If I'm not mistaken, at the time of this command being queued the current file should already be sent in the comm layer, that should have a size property you can access. Take a look at the startFileTransfer method in src/octoprint/util/comm.py. You could then also rewrite that command/add the size parameter with a queuing phase hook.

Take a look at http://docs.octoprint.org/en/master/plugins/hooks.html#octoprint-comm-protocol-gcode-phase

[bbatchelder]

Your first example. Here is a transcript of a session.

> M601 S0
< CMD M601 Received.
< Control Success.
< ok
< 
> M21
< CMD M21 Received.
<
< ok
<
> M601 S0
< CMD M601 Received.
< Control Success.
< ok
<
> M115
< CMD M115 Received.
<
< Machine Type: dremel 3d idea builder
< 
< Machine Name: DREMEL
<
< Firmware: v1.3 20150413
< 
< SN: 2d001a-33334714-37343030
<
< X: 230  Y: 150  Z: 140
<
< Tool Count: 1
< 
< ok
< 

I'll check out that link. Thanks!

[foosel]

Oh my deity, that is awful. That halves reachable transmission speeds. At least you have a full speed usb connection apparently so it shouldn't impact reachable quality, but I honestly have no idea what devil rode them to design that into the protocol O.o

That makes things more complex... Either you send an M601 for each sent line from your serial replacement, but then you'll also need to filter out every second ok that comes back pour stuff will get confused.

Or you turn every sending command into two via a hook (tricky for stuff like the rate limited M105 queries), but for that those first need to support returning multiple commands for one input command, which if I remember correctly is available in the devel branch but not maintenance.

[bbatchelder]

Yeah, it seems so braindead that I want to try and talk to an engineer at Dremel / Flashforge before going too far down this path. So far the Dremel folks have been very cooperative.

[steddyman]

I'm also an experienced developer (not a lot of python experience) with a FlashForge Dreamer and I too would like to get this working. Very happy to help collaborate or help on this one.

[bbatchelder]

Hey Stephen, I'm probably a week and half or so from having something ready to check in.

[steddyman]

Great. I am really look forward to checking it out. I have quite a bit of embedded systems development experience too so give me a shout if you need anything.

[steddyman]

Looking at the Machine Control Panel output from Simply3D it does not repeatedly send the M601 command prior to every command. Perhaps this is something unique to the Dremel which does not apply to the Dreamer. Also if I generate a gcode file for the Dreamer and look at it, it does not contain this command.

I am intrigued to know why the people who say that Octoprint doesn't support the Dreamer because it used non-standard protocols actually mean. Do you mean the GCODE is not standard or do you mean there is something different about the USB comms?

[markwal]

@steddyman non-standard means subtly different in many ways and generally undocumented though Dremel looks more cooperative than FlashForge.

To list a few off the top of my head:

1. It isn't using FTDI style serial. This is a good thing generally because it has a faster microcontroller and better pathways with more bandwidth. Problem is, undocumented yet different than all the RepRap printers out there.
2. It is using some g-code definitions that are reminiscent of MakerBot, but... once again unclear in totality which ones plus MakerBot and RepRap divided company some years ago and so things are different.
3. EEPROM settings for things like acceleration which look like they have internally (just going off of forum posts here), are no longer exposed via the protocol because they don't use x3g any more, but they didn't wire up any g-code instead (or if they did, they aren't telling anyone).
4. Other things I'm sure because it doesn't work with OctoPrint, Pronterface, Cura (as host) etc.

Really the burden is on FlashForge to describe the differences, not on the open source community to discover them all. But since they seem unwilling, it's up to people who actually own the devices to do all the reverse engineering because only people who actually have the device in front of them can do the reverse engineering.

[Cyphrz]

I may have a contact with dremel. (My company bought about 1,500 of them from dremel.)
Any questions you guys have that you want me to ask?

[steddyman]

Thanks, but not a lot clearer on this one.

So it isn't an FTDI chip, so what, it is still a serial protocol. It is implemented using the onboard STM microcontroller (which has hardware usb support) which is documented here:

http://www.st.com/web/en/catalog/tools/PF257882

Also, why does the open source community feel that FlashForge are trying to keep information from them? The full GCODE API documentation for the Dreamer is freely available on their website:

http://www.ff3dp.com/media/wysiwyg/Documents/Flashforge_Gcode_protocol_open_.zip

[markwal]

Ok. How about this way. They took open source licensed software changed it to be incompatible (it doesn't work) and in spite of the license requiring that if they redistribute the software in binary form they also have to produce the modified sources. They claim they rewrote the whole thing despite evidence to the contrary.

Plus, that documentation is minimal and just sufficient enough to notice that they've decided to be different for the sake of being different.

But the kicker is, you need a dev to make a shim. And that dev needs to both care (it has wifi already) and own one (so he can test). So if you're that dev, you should be able to answer the question yourself. If it already works, great, they made it compatible, if not, then you have work to do. I suspect you have work to do. Which is the point.

[steddyman]

Hi
Are you making any progress on this conversion?

[gretel]

@markwal i don't think that blaming some vendor will be beneficial for this project. lots of the printers are not-so-opensource but supported by octoprint nonetheless. 💵
@Cyphrz how about talking to marketing at dremel and let them discuss this issue with their engineers? 😏
@steddyman guess the 'partial' in regards to the documentation means that they don't document g-code in general, but the vendor-specific codes. 📚
@bbatchelder you have been working on something, did you commit anything yet? 👀

got a dreamer with firmware 2.4, a simplify3d license, and some engineering skills. so if applicable i'd be glad to help at some starting point. 👷

[markwal]

@gretel I know, I'm the one that did the work to support the FlashForge Creator Pro. I was just answering the question.

[gretel]

@markwal in comparison to the creator the dreamer has it's less open aspects, we can agree on that. btw, thanks for your work!

[jchristman75]

Any progress on this, or a repo where I play with the code, as it is?? thanks.

[foosel]

Just for some update on stuff that changed in OctoPrint that might maybe help:

• checkums/line numbers can be be disabled fully now via config.yaml (feature.neverSendChecksum) and
• 1.3.0+ allows configuration overlays by plugins through __plugin_settings_overlay__, so a plugin like the GPX plugin by @markwal or a FF Dreamer plugin could simply set this setting accordingly.

[slohmaier]

I made a small webapplication for monitoring and sending gcode commands to the Flashforge Dreamer using llibusb1. Although its nothing production ready: https://github.com/Noneus/FlaMo But I'm currently using it to monitor my Dreamer and send M25 if something obvious is going wrong on the webcam :)

With the stuff I learned there I want to make an OctoPrint Plugin: https://github.com/Noneus/Flocto

[KevinCathcart]

So just to clear up a few things.

The way the Dreamer and Idea Builder, and other new FlashForge models (like the Finder, or the Guider) works is as follows:

All gcode commands over usb or wifi must be preceded ~ and must be followed by "\r\n".

When you first start communicating, you use "~M601 S0" This makes the system start to listen to commands on the USB port. (It looks like "S1" is needed when communicating over Wifi) It replies with "Control Success." (and the normal "ok", of course). It is absolutely not necessary to resend this command with every new command. However the connection times out if idle for more than 5-8 seconds. Idle here means that nothing has been sent. That fact that you are waiting on an "ok" does not make you non-idle. A simple wait-to-reach-temperature command will time out under the simplistic precede each command with "~M601 S0" approach.

When you time out you get back "Control Released" followed by "ok" just like you sent a command to close the connection. This means the simplistic approach would wrongly think the temperature was reached when it was just a timeout. The reason for this weird behavior has to due with the printer's unusual command buffering system.

These printers use a command buffer on the printer side. However this command queue is very different than any others. First all all all commands can be buffered except emergency stop and the commands that return information. When you send any command you get back "CMD __ Received." For any bufferable command, once you get the received message you may immediately send the next command. Well except when you get back "buffer full" along with the received command. In that case you must now wait for some of the "ok"s to come back from the buffered commands.

While the buffer is full you can continue to send the unbuffered commands like "M105" or "M114". Obviously you want to parse the outputs of these commands all the way to the "ok" line. That said, I'm pretty sure it is possible that the ok line from one of the buffered commands will appear after you send the "M105" and before you get the result back from M105, so you need to handle that right. I suspect it is guaranteed that the output for these informational commands will immediately follow their "CMD M___ Received." lines so you know to parse from that line through to the "ok".

The fact that you can always send the status query commands at any time (except perhaps during uploading of a file) is why the timeout is so short. The flashprint software will query at least once a second.

These printers are really designed to only use the usb cable for uploading gcode to the sd, monitoring status of prints, and manual jogging etc. The printer will let you start a print from an sd card while printing via usb! So the whole premise of using OctoPrint is certainly not a a designed scenario. If after reading all the above anybody still wants to try to make this work, I've added some additional important things to note below:

Some special notes: "~M23 0:/user/filename.g" selects and starts printing. There is no need to send M24 to start the print (but doing so should be harmless. )

Yes that path does have a numeric drive-letter like syntax, which i guess is intended to allow addressing a plugged in usb stick too, using "1:", but I've not tested that at all, and it may not be implemented. (The command does seem to work fine for the main card if you leave off that "0:"). Also all gcode added to the built-in sd card should be placed in the /user/ folder, so it can be seen in the printer's gui.

Listing files from the sd card does not seem to be possible. Deleting files from the sd card does not appear to be possible.

The wait for hot-ends to reach temperature command is "M6". Command "M7" is the same but for the bed temperature. Both default to a 600 second time limit. It is not immediately clear if M119 is supported. It is certain not used by the flashforge slicer.

Hope somebody finds this useful or at least interesting.

[jchristman75]

Thanks Noneus. I will have to tinker with this. I have a PowerSpec Ultra printer, which is uses the Dreamer firmware. The vendor ID is still FlashForge, but the device ID is 0x00ff, rather then 0x0001.

In any case, I was able to connect to the printer as a preliminary test. I would love to get this working in Octo.. Thanks for the great work. I will see if I can figure this out enough to give you a hand.

[edit - typo]

[thomazff]

@thomazff I've been making some changes (upload to SD card and kick off a print, autodetect printer model) but have not committed them yet. Meanwhile if you want to try and talk to the Finder, then in octoprint_flashforge/flashforge.py change:
self.deviceid = 0x0001 # Dreamer # self.deviceid = 0x00ff # PowerSpec Ultra
to
#self.deviceid = 0x0001 # Dreamer self.deviceid = 0x0007 #Finder # self.deviceid = 0x00ff # PowerSpec Ultra

Had just made almost that. I replaced the Dreamer for the Finder. But I just get this

With GPX
Unexpected error while connecting to serial port: AUTO IOError: 'GPX plugin not able to discover AUTO port and/or baudrate. Please choose specific values for them.' @ comm.py:_openSerial:2611 (hook GPX)

Without GPX
Changing monitoring state from "Offline" to "Detecting serial port"
Serial port list: []
Changing monitoring state from "Detecting serial port" to "Error: Failed to autodetect serial port, please set it manually."
Failed to autodetect serial port, please set it manually.

[Mrnt]

Try turning the GPX plugin off in the Settings/Plugin Manager - I was testing with it off.

[Mrnt]

@thomazff Sorry, I see you tried that. Did you make sure the FlashForge plugin was enabled, does not have a typo after your change causing a syntax error, etc? There are a bunch of log messages in the code so if you turn on debugging (Settings/Logging) for octoprint.plugins.flashforge and set to "DEBUG" it might help.

[Mrnt]

@thomazff I've uploaded a new version that should support the Finder if you want to take a look.

[thomazff]

@thomazff I've uploaded a new version that should support the Finder if you want to take a look.

Thanks for the support.

Now gives the following error

Unexpected error while connecting to serial port: AUTO USBErrorAccess: 'LIBUSB_ERROR_ACCESS [-3]' @ comm.py:_openSerial:2611 (hook flashforge)

[Mrnt]

@thomazff sorry for the delay been a bit busy but should hopefully have more time now. It looks like a permission issue - are you running Octoprint on a Linux? I saw this which may be relevant: vpelletier/python-libusb1#34

[Mrnt]

@thomazff I finally set up OctoPi (which is painfully slow for me on a Pi B 1 or 2), previously I was using my desktop as I was in "proof of concept" mode. It looks like OctoPi/Octoprint on Linux does not automatically have permission to access the USB interface of the printer, which caused the connection to fail with LIBUSB_ERROR_ACCESS and before getting any useful feedback about the device in the logs.
As of 0.1.3 it should now throw an error when trying to connect in the case of a permissions problem and indicate that a udev rule needs to be created with instructions on what the rule should be and where it should be created.

[unk1nd]

Is there a repo for the current work for the plugin I can contribute to?

[Mrnt]

@unk1nd Yes! Please feel free to help/contribute/report issues: https://github.com/Mrnt/OctoPrint-FlashForge

[jimprince]

@Mrnt Just as an FYI, your link as typed is correct but clicking on it takes you somewhere else. the corrected hyperlink is: https://github.com/Mrnt/OctoPrint-FlashForge. Thanks for posting the link, it led me where I needed to go. :)

[Mrnt]

@jimprince Thanks for that!

Most recent release OctoPrint-FlashForge (v0.1.15) addresses an issue with not being able to connect to printer after it is recognized (or not uploading to SD card after connecting to the printer), due to the USB endpoints being hard coded which I think may have been an issue for someone on this thread.

Plugin should be able to connect and upload SD on FlashForge Dreamer, Finder V1 & V2 and PowerSpec Ultra but I need testers for Dremel and newer FlashForge printers.

[jimprince]

I’ve got a Dremel 3D20 and am just getting stuff setup today, I’ll report what I find.

[jimprince]

Dremel 3D20 is able to connect and upload normal Gcode to SD, after which the print starts itself. The estimated time remaining shows something non-sensical, but this is very functional for me. I did not need to install the Dremel 3D20 software in Cura or export to *gdrem format which is my process when I save to SD locally and then walk the card to the printer. I was unable to upload gdrem files, which is just as well as I'll probably stop using the SD to transport now. I did try the normal "upload" button the first time because, well I don't really know why. I guess just to see what happened. The printer and octoprint hung pretty bad and both required restarts to get going again. The printer is currently finishing it's first remote print and I'll be grateful not to have to run up and down the stairs as many times.

Thanks for putting this together!

[Mrnt]

I might be able to get the *gdrem format to be accepted if that is helpful? Is that the format that includes the image in the file?

Yep normal OctoPrint upload button will not work yet, but I think Im close - looking for volunteers to test that.

I think you should be able to upload direct to SD card from Cura (via OctoPrint) if you point Cura at the OctoPrint server in the Cura printer settings.

[jimprince]

I'm not super fussed about being able to use gdrem files and tbh it cuts out a step I've been doing for so long I've forgotten that I'm doing it. I believe that format does have the image in the file, but I'm uncertain how it goes about that.

Happy to be a guinea pig when you need to test the upload button. :) Online print status would be incredibly helpful.

I'll have to give the upload directly to SD from Cura a shot, waiting for this print to finish.

[jimprince]

I am getting this in response while printing, which is probably why the time to finish looks so weird on the printer itself.

Printing directly from Cura does work, but it does not work with the Dremel profile that I installed from the marketplace (it won't let me connect to octoprint with that machine profile). I just created a custom machine and then was able to get octoprint to connect. I had to remove the start and end Gcode from the profile as I was getting communication timeouts when they ran. After removing the start and end gcode it looks like it's working like a champ.

I'll put any issues I find in the actual repository you've linked to.

Edited for clarity

[jokedu]

i just installed it on a raspberry pi 3 yesterday.
i also bought the night vision camera for the raspberry and 3d printed a case and a mount, i currently own a dremel idea builder / 3d20 and have been getting problem connecting it to octoprint but i am later going to try manually selecting the usb port with this:Connect to your OctoPi/Octoprint device using ssh
2) Type the following to open a text editor:
sudo nano /etc/udev/rules.d/99-octoprint.rules
3) Add the following line:
SUBSYSTEM=="usb", ATTR{idVendor}=="2a89", MODE="0666"
4) Save the file and close the editor
5) Verify the file permissions are set to "rw-r--r--" by typing:
ls /etc/udev/rules.d/99-octoprint.rules
6) Reboot your system for the rule to take effect.

This is a copied text of the terminal in octoprint's website, i managed to get this text by connecting my printer and installing the pluggin of flashforge and dremel.
Does anyone has a better solution or how did you managed to get it running and connected?

[jokedu]

and all of you with a dremel 3d20 how do you get prints to stick?

would be kinda cool to know what different people use

[Mrnt]

i just installed it on a raspberry pi 3 yesterday.
i also bought the night vision camera for the raspberry and 3d printed a case and a mount, i currently own a dremel idea builder / 3d20 and have been getting problem connecting it to octoprint but i am later going to try manually selecting the usb port with this:Connect to your OctoPi/Octoprint device using ssh
2) Type the following to open a text editor:
sudo nano /etc/udev/rules.d/99-octoprint.rules
3) Add the following line:
SUBSYSTEM=="usb", ATTR{idVendor}=="2a89", MODE="0666"
4) Save the file and close the editor
5) Verify the file permissions are set to "rw-r--r--" by typing:
ls /etc/udev/rules.d/99-octoprint.rules
6) Reboot your system for the rule to take effect.

This is a copied text of the terminal in octoprint's website, i managed to get this text by connecting my printer and installing the pluggin of flashforge and dremel.
Does anyone has a better solution or how did you managed to get it running and connected?

@jokedu I know it is a bit frustrating, but you only have to do it once to enable USB access. I suppose it may be possible to get the plugin to create/modify the file automatically, but right now it seems like the development effort would be better spent extending the plugin functionality/printer support.

[eduncan911]

and all of you with a dremel 3d20 how do you get prints to stick?

would be kinda cool to know what different people use

OT, but...

We don't have any issues with prints sticking and just finished the first roll of filament.

Using the stock sticker pad it comes with for now, though I've been eyeballing a PEI pad replacement. Still got the 2nd pad on the bag. So will use that first.

All we do is manual bed level during each print. Insert the orange paper thing and adjust the bed to where it barely scraps that orange pad.

[eduncan911]

@Mrnt

I might be able to get the *gdrem format to be accepted if that is helpful? Is that the format that includes the image in the file?

Yep normal OctoPrint upload button will not work yet, but I think Im close - looking for volunteers to test that.

I think you should be able to upload direct to SD card from Cura (via OctoPrint) if you point Cura at the OctoPrint server in the Cura printer settings.

Count me in as a tester as well. Just ping me here on GitHub with @eduncan911

[Ve2mrx]

If you need testing, I have a Dremel 3D45, plenty of Pis and some professional troubleshooting experience.

[Ve2mrx]

I have a disused laptop with Windows 7 installed on it. It may already have Wireshark on it... If that can be useful, ping me!

[eduncan911]

Oops, commented on the wrong thread.

As far as I am concerned, this thread with the 3D20 (aka Ideal Builder) can be closed as it's fully supported with this plugin:

https://github.com/Mrnt/OctoPrint-FlashForge/

[Ve2mrx]

@eduncan911 I am subscribed to both :-)

[foosel]

As far as I am concerned, this thread with the 3D20 (aka Ideal Builder) can be closed as it's fully supported with this plugin:

https://github.com/Mrnt/OctoPrint-FlashForge/

Indeed! 😅