New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Request] Stop changing serial commands and messages on a whim #2014
Comments
I'm with @foosel here. @thinkyhead Do NOT. EVER. EVER change the error messages. I know the current situation sucks, but if you change things without talking things over with the host software developers you will cause chaos. @foosel Shall I give you rights on this fork so you can revert stupid changes like this and wack people on their heads when they do this? |
@daid I'm tempted to say "Yes please" just to reduce the amount of headaches the recent events have been causing me. |
+1 vouching for control by @foosel from a host perspective. |
+2 ! |
Is there such open source protocol for serial communication with a 3D printer? Can it be based on a existing RFC or ISO? If not, who should define the protocol for communication between the open source firmwares and proprietary hosts? The firmwares coders? Hosts coders? An independent committee? Will he ever had any effort to create a standard serial messaging. If yes, what can we learn from it? I'm just thinking out loud... |
FWIW, I got so fed up with the lack of a proper protocol and protocol definition and the constant problems that are caused by that situation that I recently started working on a baseline to get one crowdsourced, together with @jnesslr . Hoping we'll have something to go public with soon so that we can all make the world a better place in that regard inside the RepRap community. Focus is the open source world though. IMHO proprietary projects can go play in their own sand box for all I care. |
Proprietary hosts can figure it out on their own. But we are all FOSS devs here I think. And we don't want a gazillion different versions of the protocol to support. |
+1 for @foosel FWIW the "spec" does say debug information should be prepended with // and the should just be displayed on the host and ignored, why don't Marlin developers use // for the debug output... http://reprap.org/wiki/G-code#Replies_from_the_RepRap_machine_to_the_host_computer |
@wolfmanjm that "spec" was written after the fact. And does not even match the Marlin implementation used the last 3 years... |
@daid That maybe true but I think it is a good idea, and FWIW Smoothie is now using that concept. |
+1 Regards/Saludos, Ernesto
|
@foosel Can we take a look at your protocol? How should it work? A simple protocol definition or a complete implementation, abstraction, supporting all processors, 3D printers, hosts? Should it be only for serial communications? A Web API or any other board connectivity? A separate repository or let firmware coders follow the guideline? Still thinking out loud... |
I love it when the system works. Good to see everyone too – hi! |
Power to continue playing solution how to write |
@clefranc As soon as it's in a presentable state, right now it's mostly creating an outline and nailing down some basics like ABNF syntax and basic communication specifics. The idea is to not create a new protocol, it is to finally formalize what is already in place today in regards to the communication that is taking place over our printer's serial interfaces. Of course, since we don't have an agreed upon standard (otherwise nothing would need to be done) that will mean that a compromise will need to be found on things that currently differ between the several implementations out there (e.g. format of I wouldn't limit it to serial interfaces -- if a printer speaks TCP natively, that shouldn't matter (although things like checksuming wouldn't be necessary in such cases since thats already taken care off by the underlying transport layer). |
+1,2,3,4, and 5. That recent G28 change debacle had me kind of angry to be honest since, out of the blue, my printer no longer raised up before printing which meant head crashes. That could be very costly on the Mendel since it's all solid mounted and a PEEK hotend, luckily I have a more flexible printer to test things on so head crashes just bounce off the springy bed. |
Changes should not be made to the code that is being distributed without having it tested first. I don't understand the unwillingness to roll back the code to the point where everything worked. I've been told my time would be more 'productively used' to help debug the sick and broken code than to complain about the practices that got us into this state. Mean while, we see threads saying "Testers Needed!" Good luck with that! My printer can't even print using the current broken code. |
To be fair, the code is being tested, it's just that the main branch you see when opening marlin on github happens to be the development version. I think it should probably be hidden away in a dev folder and the actual "stable" 1.0.2 release be the main code you see when you first open it. I just wish I knew how to code so I could actually help in some way, other than sitting here complaining about stuff I don't like. I feel useless. |
Given the amount of permutations due to the sheer number of |
I agree that having a "Release" branch which is always at the last tagged commit, 1.0.2 in this case, and marked as the default branch in github, would help a lot of users. Developers will be familiar enough with git to change branches before working, and the act of doing so should put most users on notice that they're headed into the untested wild. The tagged "releases" at the top still serve a purpose, to go back to prior releases, but people that just download from github and try compiling will be doing so against the last tagged release instead of whatever latest commit happens to be. @boelle -- thoughts? @Roxy-3DPrintBoard This may address what you referred to as a reluctance to roll back. Unless a commit is wholesale "bad", it shouldn't be rolled back because there's something good in there worth keeping. A new commit fixes the problem and continues to move the project forward. But that's only true of a "Development branch" where this sort of thing is supposed to happen. A "Release" branch would only ever contain the tested and vetted releases that Marlin developers want the world to be running. The current Marlin project is a Development branch, and tagged releases which general users can download for use. Our users don't seem to be catching on to that if they're getting angry about individual developer commits with limited testing (at most their own machine). They're probably cloning the Marlin github repo ("Development branch") and then compiling whatever code is the latest today. Head crashes ensue. |
@gregrebholz if a single commit contains more then 1 change then the commit is flawed anyhow. See for example the change that triggered Gina this time. It was a hidden message change inside a change that added a debug flag... |
@daid sure, from a philosophical standpoint a commit should be to accomplish a single goal. That can be many lines of code, however, with only a single problematic line. Wouldn't you rather take a new commit that fixes the typo (say found a day or two later) over "reverting" the whole commit (rewriting the revision history for people that have forked) and then doing a new commit that fixes the typo? In the context of a Development branch which is trying to always move forward. |
@gregrebholz depends on who made the commit, what the commit is actually trying to accomplish. And if I want to be an asshole that day (which is usually a yes) |
Pretty much... Everything with regard to the Broken G28 and Broken G29 has been with the attitude "Let's try this." or "Let's try that." There isn't much of value. Rolling it all back and starting fresh makes the most sense. And really... That is the whole reason for using a Revision Control System. If there is some thing else of value, well... It can be added back after we roll things back to the point they work. |
not any thoughts... i have just been trying to keep head and tails in the issues and not worried much about anything else |
but a +1 from me to give @foosel collab access. maybe even hope that she can help out getting a few bugs sorted |
My issue and "anger" came from the suggestion that they had "fixed" G28 to "work as it should" when in fact if you look at CNC references G28 doesn't work that way at all. http://www.cnccookbook.com/CCCNCGCodeG28ReturntoReference.htm
So the claims that it was now somehow fixed, were in fact even MORE broken. Not to mention no consideration was ever given to every host program out there would need to be modified in order to work with the "fix".
Then perhaps they should ask for more testers before committing it to the main code repository. Just sayin'. |
Because in a cnccookbook it is written that a G28 X5 should move, it is broken in Marlin? |
I think there is more than one interpretation of G28 if you read the page up to the end:
But: Shouldn't we stop this flame war and go back to work?
|
I guess host with support for newer protocol versions talking to firmware still taking an older version is going to be a much more common scenario than the other way around though, especially since updating the host is trivial (and can even be done automatically) compared to the process of compiling and flashing a new firmware version, hence the suggestion to make it a one sided adaptation. From what i understand of firmware development it will probably be a challenge to cram multiple protocol versions into the limited amount of space there is for the firmware so that in the end it will probably boil down to just one version being supported anyhow due to platform limitations. So basically we want the same, but I am pessimistically assuming that it will result in one side always only supporting one version anyhow ;) Just realised: why lowest common? Highest common would probably make more sense, no? |
@foosel If the host implements the new protocol before the firmware then that would be right. But what if the firmware side comes up with a new Idea? You are right that having two protocol parsers would be too much overhead for the firmware. But most of the time it will be only small changes from one version to the next (as the discussed subscribe to changes feature) so that is nearly no effort for the firmware to support both versions. I would suggest a mechanism to send a list of supported "extension" to the host. This way it would not be necessary to increase the version number for each small enhancement. But then there is already a G-Code for that on the reprap Wiki page, So I will not suggest it. And yes I would also go for the highest common protocol version. And if you read closely nophead said "the lowest of the two" that would be the highest common, right? |
If the host supports version h and slave version s then the protocol version to use is min(h,s) i.e. the lowest of the two, as I stated. However that could be described as the highest common version, i.e. the highest version that they both support. In terms of firmware not wanting to support old versions it could just reject hosts that were too old in its opinion, just as the host could refuse to work with firmware that is too old. |
Sorry, I somehow misread it as "the lowest common" instead of the minimum
of the two, hence my confusion.
|
Another option is for the host to ask the firmware what optional capabilities it has, then the firmware can reply with a list of all its add-ons, by name. (For example: So, some of the good ideas here worth doing soon…
|
I think we should split what is often called protocol here into 2 parts - protocol and features. First I'd like to narrow down what I think we should define as a protocol. A protocol should be the basic rules to communicate. And yes, this protocol should in deed have a version number, but that will only change rarely if we define it as communication agreement without all the command implementations. This is already done in Repetier Firmware which has by now 3 protocol versions, mainly because I added new value names to the binary protocol and sending new values to old parser will cause communication errors. So with protocol is what we need to have a communication going. This means:
The other area is features. Users can compile or not compile in features, so a version does not help. The idea to have a command returning a list of implemented features would be a proper solution. Here again we could have a version problem if features get changed somehow in a incompatible way. What would be possible is a feature list like Why 2 numbers? First number is oldest version feature is still compatible with and second is current version. That way if host implements version 1 and some firmware developer adds a feature e.g. new gcode parameter, but with a default that it works also without it, it is still usable for the host that knows versions 1 and host knows it is compatible. Same for added responses. As long as old answer parts are the same there is no need to increase oldest version number. We should assume hosts parse only return values they know ignoring unknown type:value parts. I do not assume firmwares will share same names and version, so this is a per firmware definition. I guess to make all this work each firmware would also need to have a webpage describing all implemented protocols and feature sets including versions, parameters and version introduced and response values. That is more then the reprap wiki or a simple list of supported gcodes does. It means real documentation at one place for each change and a description of the current state (which would help @foosel with the wiki as well I guess). That will be the hardest part as it is no secret the programmers normally hate to document and do it only when REALLY needed. And especially with a system like github where many can commit, there must be someone documenting all changes having a change in communication. The price we get is a much nicer user world where hosts disable unsupported featured them self and enable others magically giving a much better user experience. |
But not without discussion please. If Marlin does its own thing with these points, nothing is won, once again. @repetier I'm not so sure about separate version tracking for specific features. I'd rather see protocol extensions depending on specific protocol versions. Otherwise it will get out of hand quite fast. The extended functionalities that are interesting to more than one printer manufacturer out there should be properly specified as well as the core functions, otherwise you'll get 100+ variants again (although it gets easier of course if they are versioned, but still)... I'd actually prefer to see the means of limiting the wild growth a bit and canalizing it into sane ways. I could imagine something like vendor extensions that could later be upgraded to full protocol extensions. Think basically every RFC out there.
The idea is basically that the protocol definition comes first, so the only thing that would be necessary to document by the firmware would be "now compliant to RepRapCode v1.2.4". This is exactly due to the reason that nobody likes to document their stuff and hence we are in the mess we find ourselves in. So from my point of view: First and foremost, the basic communication foundation has to be laid. So as you said, basic request/response syntax, error handling, behaviour in case of unexpected/unsupported data, push messages, protocol handshakes, command structure, where vendors can hook in their extensions without colliding with the core (which can then be safely ignored unless supported explicitely), stuff like that. Then we can add core commands (M105, M117, G0, G28, all that stuff you really NEED to operate a printer's core functionality -- so no G29, no M117, etc at first, the real pure basics). And when THAT is stable it makes sense discussing stuff like auto leveling, filament run out detection, etc. Also, one thing I forgot to answer to earlier from @JustAnother1 :
First of all, new ideas shouldn't cause the communication to become incompatible. There should be well defined extension points to allow for situations just like that without having to switch protocol versions, like you said. But more importantly, I was actually talking about real world user operation. I know everyone partaking in this thread is probably constantly flashing their printers to the bleeding edge versions of the firmwares. But our users don't. I'd go even so far as to say that most of them NEVER do. And this is a really big problem in and of itself (e.g. I still have work arounds in OctoPrint for a lot of problems I have since PRd fixes for in Marlin since I can not rely on the user actually updating their printer's firmware, since e.g. on Printrboards it's a huge PITA and even if it's as easy as just "click here" most users just won't "click here" unless basically forced to). So the norm that we should expect is that the host is probably more up to date on the evolving protocol versions than the firmware. And this is what we need to focus on, leaving a way out for the developers of course. Which both parties exchanging the max. supported version number and then using the lower value of that is (plus extension points for developers), so everything's fine. |
Usually because somewhere on nearly every device there's a warning something like Flashing your firmware can be dangerous and lead to issues and should only be done when necessary so people get scared off when they see the word firmware and flash. |
Exactly my point. |
@foosel So we already agree with the protocol stuff. That will also be the first basic part in your protocol reference and the current state is known and working. So at least for the ASCII part we should see when you are finished that we give it a name to specify for each basic firmware supported. Everyone then adding features must at least keep himself to that minimal protocol. Feature protocols are the biggest problem I guess. Not that I need to stick to my versioning scheme, it was just an idea. The problem is regardless of what we do or would like, there will be changes even to defined protocols. Maybe not from us, but there are sooo many forks with individual changes where programmers even might not know about what we want. Not that I intend to implement one of these island solutions. Main version have already enough changes to follow. So from my side we can start with something like a string list AUTOLEVEL AUSTOSEND_TEMP and next upgrade then adds AUTOLEVEL_EXT1 or what ever and removes AUTOLEVEL if it gets incompatible. That way we do not look for version but named features in a set to decide handling (maybe internally always adding a firmware name to distinguish). Most important is to have a normalized way to find out about features and having a documentation what the implementor means with er.g. AUTOLEVEL which can easily spread across a number of implemented gcodes with defined behavior. As a first step I'd like to propose for feature negotiation |
A standardized "not supported" response from firmwares for any unrecognized G/M code, or a recognized code that doesn't parse as expected, could be a beneficial way for hosts to react intelligently to "surprisingly old" firmware without a detailed per-feature versioning system. As long as we're still brainstorming (though there's a lot of shooting down going on already too, so...?) - what if updating your firmware became something that hosts could offer to users? Certainly this happens with commercially developed products with MUCH smaller ecosystems ("an update is available for your widget!"), but if one host-firmware pairing started doing this, the hosts and firmwares that didn't play along may fall out of favor... I think it would be very popular with the general user base. This would require either (a) popular firmwares like Marlin to develop an XML-based feature configuration file that hosts could reasonably create/update from user input (Config > My Machine menu, or a wizard, etc.), and feed to a host side build process. This configuration file could also be beneficial to the host for understanding the features and limitations of the machine; or (b) popular machines could publish versioned .hex binaries for their mechanics and perhaps a few permutations of user replaceable parts like hotends and thermistors. Hosts would check a web URL and offer up the latest .hex file that matches a user selected profile of what machine is being managed. There's little risk of "bricking" a controller with the Arduino boot loader intact, but of course there is risk of head crashes, thermal runway, etc. if the configuration doesn't match the mechanics, pinouts, etc. These .hex files would be the responsibility of the machine manufacturers to maintain, and users would be admonished to only select a profile if their machines is exactly as the one described. But in total, this would be the most likely way to get users keeping their machines current on firmware... and this is a good thing for safety - how many machines have no thermal runaway protection at all because they have a version of Marlin that's 2 or 3 years old? Just a thought. |
@gregrebholz A thought I also had and planned as a plugin for OctoPrint, which @Booli in fact is already working on (the flashing only part for now as far as I understand). Add to this some way of detecting if new official firmware releases are out (could easily be done via the Github release API for example, OctoPrint already uses that for its new self-updating feature) and some compilation service online (or maybe built into the host if push comes to shove) and you have basically my thirty thousand foot perspective of were I want all this stuff to go usability wise in the mid to long run. |
For Repetier-Server we are also working on auto updater. I have already a first idea how vendors could offer updates, also it is not finished completely. I have added the idea at the bottom. We can only expect this to work with printer vendors supporting updates and offering the required hex files. Only for these printers we have enough copies of the same hardware. Of course we can always add a hex uploader for any file but all these unique self made experts will always need their own config. But at least we could catch most sold printers if vendors would implement the one or other type of update mechanism all hosts could use. Most vendors are always very busy so it's unlikely they will support multiple types of mechanism. So if they do, it will with highest priority for the host they bundle with. Big vendors maybe with other popular ones, but I guess it would be better to agree on a standard for all. So if you have anything comparable for Octoprint I would propose maybe a talk about this (maybe not in this thread), so we can make a more common standard in first place before we start implementing several versions to get best acceptance. Till now I have only my idea below, no implementation so I can easily switch to other notification methods. ------- Idea how autoupdater could work -------- Automatically detect expired firmware and offer firmware updates. Stage 1: Detecting printer firmware We need to detect the currently used printer model and firmware version. We also need a url to retrieve M333 ; Retrieve firmware informations The base url must be a directory on a webserver. It must contain a file named That file has the following layout:
The file can contain versions for several printers as long as each printer has a different model name. To allow users to go back to older versions, we allow definition of multiple versions. Each version Versions are up to three numbers separated by a dot. No chars allowed. It is allowed to omit 1 or 2 parts of the |
@foosel I agree completely. If there is an "extension point" or whatever so that the lonely cowboy coder can come up with a new extension, implement it in a firmware and then patch a host and use it to show that it works then that is fine with me. I think the also mentioned safeguards(version check, ignoring unknown commands,.) will make sure that this will work. I also fully agree that we need a discussion first before we start writing down all the details. And I agree with @repetier that this thread is not the best place for such a discussion. I think it would be good to not only discuss this on neutral ground, but to also invite the whole community to take part in the discussion. Therefore we would need an either a well known place or send invites once the place has been decided. So in the interest to push this to the next level: What would be the best "place" to have the needed discussion? reprap Forum / reprap Wiki / http://reprapcode.haz.wiki/ / a github repository / ... ? |
My initial thought was to add disqus discussions to the reprapcode wiki pages so you can basically discuss the stuff where it comes up (so single commands, handshake etc). This has since happened, to there are now comments in place that should basically allow everyone to take part in public discussions right on the page. It would probably make life easier to have the discussion at the same point as an attempt at standardization itself, so from my point of view we could just move there. |
Just remember, string space is tighty tight tight on these boards, so of course the less verbose the better. |
Hey guys, this is very exciting and I wanted to be sure you know that we (MatterHackers/MatterSlice) are at interested in participating and helping with what is happening here. It looks like there you are already having a very good discussion. I think there are a lot of good reference libraries that server their industries well that we should look at as reference. The one I'm most familiar with that seems to fit our needs is OpenGL. The main qualities that it has that I would think we want are:
We will also probably need to define some type of governing body to arbitrate if we want this to really get off the ground. Hope this helps. Great work! |
I also thought about a "test suit". The problem I see is that most commands do not have an effect that can be read back. If we send a command to set a temperature then we have to somehow check that the heater is turned on. Here a limited feedback would be possible, by reading the temperature values. When we send movements we need to check if the head really moved. Especially tricky with different federates. Was the movement done with the right speed? What about homing. How can we test if a command to home the head has succeeded. How do we check where the head is. I don't think that it would be practical do do manual tests here. Just sending commands and if the firmware replies with OK is not enough to ensure compatibility. A simulator would be a option. talking to a firmware running in the simulator and then checking if the firmware switches the right output pins. But that would be a lot of effort. So a test suit is a good Idea. I just don't know how to make it work. Do you? |
For the sake of confirming if a given build of firmware is compatible with the proposed protocol, I'd think validating send and receive messaging is exactly what we'd want to test. In this context it doesn't matter if the firmware is executing the motor movements correctly, just that it responds in the expected manner when appropriate commands are given. Having a test suite would ensure that non-conforming/breaking changes are caught early and are caught by the firmware team rather than the potentially painful catch in host software after lots of troubleshooting. |
How is |
To Whom It May Concern: RepRapCode Specification effort (linked above) has moved over to a Github project due to negative feedback concerning the previous collaboration model. @thinkyhead sorry, saw the question just now, but: |
In the future hosts may take note of the firmware version and use the |
This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs. |
Case in point, this specific change. Also the recent change to the behaviour of
G28
that thankfully got rolled back again.We do not have a proper transport protocol in place here (the GCODE RepRap Wiki page is NOT a well defined protocol definition!). That means that as a firmware you have to be very very careful when making backwards incompatible adjustments to your external interface, which your serial interface is. Host software has to work with what is there in terms of a "protocol". Changing messages like the one in the commit linked above (especially if hidden like that in an unrelated commit) can have consequences beyond Marlin.
"But the hosts do very ugly things they shouldn't be doing!" - I agree! And I hate it too! But do you know the reason? Because there is no proper transport protocol so host developers have to constantly depend on workarounds and dirty tricks to make the more sophisticated functionality requested and expected by the users work at all without them having to understand G and M commands and even touch a serial terminal (and btw - make all that work across firmwares, so not only with Marlin but also with Repetier, Smoothie, etc). Changing commands or messages on the serial interface out of the blue like this does NOT help in making our job easier, so please be a bit more considerate! At the very least stop hiding stuff like this in the mix and start documenting it properly.
Thanks in advance!
The text was updated successfully, but these errors were encountered: