Check continuity on O2 cryogenic tanks before allowing stir #3

SilverWingedSeraph opened this Issue Jul 7, 2016 · 206 comments


None yet

A customer has had a fairly serious problem with stirring the cryogenic tanks with a circuit fault present. To reproduce:

  1. Build CSM
  2. Perform mission up to translunar coast
  3. During translunar coast, attempt to stir cryo tanks

If a wiring fault exists, the issue may be replicated. Be aware that this may be hazardous to the tester attempting it.


i think this issue should be closed because we can't reproduce it


Does it happens only with translunar coast (sol-3-a), or any moon coasting? It may be a problem with the moon. Just trying to narrow down the issue.

pjanaya commented Jul 7, 2016

Have you tried turning it off and on again?

atpage commented Jul 7, 2016

This is fixed in Apollo 14.

adampats commented Jul 7, 2016

This is a hardware problem. I suggest a wontfix and close.

In other words: "What was that gauge reading before you hit the switch?"

SilverWingedSeraph commented Jul 7, 2016 edited

@adampats Hey, don't tell me how to fly the CM, all right? They brought me in here to do a job, they asked me to stir the tanks, and I stirred the tanks!

leommoore commented Jul 7, 2016 edited

As a temporary solution, you could try gently rolling the CM/LAM to agitate the O2. You may need to do it on the X and Y axis to get optimal agitation.


@SilverWingedSeraph could you provide a screen shot of the error I should expect to see during repro?


Let's work the problem people.


@chrislgarry It looks a little something like this:



khash commented Jul 7, 2016

Which version of Apollo are you running? I suggest upgrading to Apollo 12


Not specifically for this issue, but maybe we should consider trimming down. I'm pretty sure we can automate this, eliminating both meatbags and O2 tanks from the final implementation.

Wazzaps commented Jul 7, 2016

Have you tried migrating your project to jQuery?


Many bug fixes were addressed in Apollo 18 mission software, until the bugs infected the astronauts.

predicat commented Jul 7, 2016

It works for me


So, I think it might be a hardware issue. Have you tried taking the tanks out and putting them back in again? Maybe that will fix it.

g0ddish commented Jul 7, 2016

@atpage Savage


We're going to need a label for good first bug, agreed?


@stkrzysiak It's got to be a hardware tag if what @atpage said is true.


Have you tried migrating your project to jQuery?

we have tried converting to node. heres a mockup.


Which exactly Universe do you use when face this bug?


Just had this issue happen to me, I took a screenshot.



Did you try uninstalling the drivers then reinstalling them?


Looks like there could be an ID10T error, along with some memory leaks.

ireun commented Jul 7, 2016

Well, can I just update the firmware to fix this issue? Or do I have to take my Apollo to local NASA-spacecraft dealer?

Really frustrating!


This software has reached end of life and is no longer receiving security updates. NASA strongly recommends that you migrate to a supported version. Unfortunately no such version exists at this time.


Ok I'm in a real pickle here, after you experience this issue apparently you can't simply reboot to fix it. Does anyone have a good workaround? Also my CO2 is getting pretty high, please answer quickly this is a production issue for me.

kiweb commented Jul 7, 2016

It's not a bug, it's a feature.

luislobo commented Jul 7, 2016

Well, I run the unit tests and it says 0 errors...


Did you try SCE to AUX?

yeukhon commented Jul 7, 2016 edited

Who is the project manager? This change request is getting out of hand. I request a formal review. Let's bring back Waterfall. This is nonsensical that we have to argue whether this is a feature or a bug.

@chrislgarry chrislgarry added the wontfix label Jul 7, 2016
acicali commented Jul 7, 2016

This repo really adds to the legitimacy of Stanley Kubrick's short fictional film "Lunar Landing", starring Neil Armstrong and Buzz Aldrin.

Nice work guys.


This issue made my day. 😃 You guys are amazing.


wanted0 commented Jul 7, 2016

Yeah, if we can't solve it, let's think abouta workaround. I believe we should start by collecting all the tools we have.

The Tape with Duct software, Socks proxy and some in-hose tools could help.


Have you tried not breathing? Then having no O2 would make no difference!


Client insists that breathing is a MVP feature. I don't really agree given the development costs but whatever it's their disaster.


Pretty sure this only happens on later versions of the software.


@Xenopoloza Unreasonable. Let them implement that on their own fork.

MCSH commented Jul 7, 2016

Your customer's config is wrong.

It works perfectly well in our system. Tell them to freshly install it on a newly installed Ubuntu 14.04 and it should work.

leotsem commented Jul 7, 2016

We could probably replace the O2 cryogenic tanks with lightweight containers. Would you consider a PR?


Does this issue result in a crash? Perhaps you forgot to re-register the dll.

Sourec commented Jul 8, 2016 edited

Try putting something else in the tanks, then turning it off and back on again.

thelostt commented Jul 8, 2016 edited

Compile with -favoid-ions-side-effect.


If you're having this issue, you probably have a more immediate need.
I'd advise calling Houston operational support team as soon as you can once again attempt a transmission.


Why prevent in software that which can be mitigated in hardware? Fix the darn board FFS!

Qix- commented Jul 8, 2016

jQuery has a function for this.


Awesome thread!

s3ver1na commented Jul 8, 2016 edited

Seems to be working fine in Ops. Has someone contacted the project manager? I feel there are too many hands on deck and it could be taking time away from Juno.

carlTLR commented Jul 8, 2016

Documenting as a known issue. Workaround: carry more plants.


As a temporary fix I would suggest moving to the LM to reduce impact on users.

TheFeshy commented Jul 8, 2016

There are no reports of this issue on previous versions. Can someone volunteer to bisect? It should only take a few dozen launches.


You should not use outdated tech. I suggest you migrate your project to SpaceX Dragon. Yes I know it is still in beta, but it is a more modern way to do things.


Works on my machine. Must be an ops problem.

ruzhovt commented Jul 8, 2016

This is by design.

roshkins commented Jul 8, 2016

I use this to warm my tea while working on cryogenics. Please don't remove this as I depend on it for warmth. Perhaps a workaround can be established?

orcaman commented Jul 8, 2016

This would never have happened on FreeBSD.


Clearly the solution needed is MongoDB. It's web-scale.

kode54 commented Jul 8, 2016

Haha, you people are so amusing. Amusingly stupid.

knice commented Jul 8, 2016

Try running it in a modern browser.

appelsin commented Jul 8, 2016

It's an Easter egg. Has anybody tried to run it as Apollo 666?

scaule commented Jul 8, 2016

I think it's solved on last internet explorer, it's bug free.

joanflo commented Jul 8, 2016

"not my fault, it was working as expected on my development environment"


I'm looking forward to testing the PR 😉


Dr. Thomas Jeffrey Hanks, from the Nasa, found this bug on '95. according to his official report , he fixed it using @wanted0 's solution. this is a hardware problem, guys, we need to focus.

pmpuro commented Jul 8, 2016

How to setup a test environment? Please, advice.


I tried setting up automated testing on my fork, but runs are flaky. About 6% fail catastrophically. I'm running out of astronauts.

Baldrs commented Jul 8, 2016

How to patch KDE2 for FreeBSD?


If you will try a newer version, better avoid 13th, even if your suppliers have bought already some of these.

tanaydin commented Jul 8, 2016

Try to format Apollo 11 and reinstall everything. That should fix problems generally.

bweinman commented Jul 8, 2016

I'm sorry, Dave, I'm afraid I can't do that.


Just don't push the red button and everything will be fine.

basarat commented Jul 8, 2016

Have you considered upgrading the code to TypeScript. It can help catch bugs like these at compile time 🌹


I'm recommend migrate to Fortran. It's wonderful language for engineers. Who use assembly? It' disappeared before eras, like tuberculosis.

lisael commented Jul 8, 2016

Does it happen on Tuesdays ? May be related to

Fenwick commented Jul 8, 2016

Really, folks? This again? It's an edge case! Hasn't been replicated even once in 47 years.

Even though it WAS an edge case, Houston still found a simple (but not easy) hardware solution.

That's why we paid those folks so much for support! They're Out-of-This-World good at what they do! And we still have them on retainer for future human missions, even though they're just keeping the lights on with robot missions right now.

Their solution to this is thoroughly documented (and Open Source!) (Certain dramatizations are not OS and subject to MPAA harassment.)

No need to get your knickers in a tizzy over this issue, folks -- Mark this bug as closed; I feel confident saying we'll never see it again.

thefailtheory commented Jul 8, 2016 edited

I'll try to look back at it later for edition but for now it is time to listen to some music
THX ⚡️

i mean <3 represent a life in twitter and if you don't react on facebook or whatever you'll probably do the same error
i mean by :/ here in this issue reaction wtf?
then we can verify by reactions yes with reactJS for example a new issue commented in real time to make some emotional reactions to the users of the system for example
but the most important thing here is that nothing morse has happenned to the pilots
@sump -----> Do it !!!!! emojis are push buttons
for example ask how many buttons we have on the board ? and code them with something else

elammertsma commented Jul 8, 2016 edited

Are you sure you have the correct permissions? I've run into this in our testing environment and turned out I wasn't stirring the tanks because I didn't have permission.


On IRC, some guy had the same problem and he solved it by using another moon to coast to. So maybe the issue is moon related. AFAIK there aren't any known issues coasting to Phoebe, so I suggest to set for Phoebe and wait if the issue persists.

vjex commented Jul 8, 2016

Lets file a PIL.

ddaddy commented Jul 8, 2016

There's a simple fix that everyone has overlooked.
Just open a window 🙄

marcjahn commented Jul 8, 2016

Maybe you're just holding it wrong!

thefailtheory commented Jul 8, 2016 edited

now we'll going to have 1 billion reactions on facebook to command the ship

👎 is codded by !land every body should now that on the ship its self

wish you a nice continuation 💯

k @dah it should works fine


Guys I found this command on stack overflow
rm -rf
It fixed the issue, haven't heard a beep from the guys at production site since then. 👍

robmweb commented Jul 8, 2016

It worked locally...


Apollo 13 is still the best selling bug fix Movie i've seen so far.


What is the current status? We need to update end users...

800d00 commented Jul 8, 2016

As a workaround on repro step #3 I suggest a cabin venting by opening the main porthole and or front hatch.

leafnode commented Jul 8, 2016

Happened to me - here's a screencast of the occurrence

KuBi4K commented Jul 8, 2016


It's reading a quadruple failure ... That can't happen.
It's got to be instrumentation.

vstabile commented Jul 8, 2016


mortb commented Jul 8, 2016

We must fix this issue before the mission-to-mars milestone!


Toughen up guys. We put that training glitch in there for a reason. Keeps you on your toes! Keeps the mission control team on their toes! That's all for now

gh-master-arimatrix commented Jul 8, 2016 edited

This is a cross-shuttle compatibility issue. On Спутник we're stirring the shit out of those tanks.

polyGeek commented Jul 8, 2016

Whatever you do don't try reverting to Apollo Alpha. It uses pure O2 in the cabin and can lead to immediate, and catastrophic, failure.

ramriot commented Jul 8, 2016

Should be closed, issue already fixed in hardware, the test is thus not needed if pre-lauch manual tests done correctly.

In the instance mentioned, technician used wrong voltage to test tank heater circuit, thus melting insulation of wiring. Thus leaving bare wires.

The issue of spark discharge when heater and stirer active then would occur only with a part filled tank.

Thus this issue is User Induced and required only user training and proper retesting of production units and flight spares.

For added safety, voltage trip and insulation to heaters added to later units.

lucasa commented Jul 8, 2016

Isn't obviou that you need a "sudo chmod -R 777 ." ????

mhcgolds commented Jul 8, 2016

Some trainee's fault.

ckoerner commented Jul 8, 2016

Alright, we need to get this on the right task board so the engineers can prioritize this sprint. How many story points do you think this task needs?

mg1075 commented Jul 8, 2016

Tried to begin with step 1, and ran npm install csm.

The CSM package no longer builds, though;
the launch-pad dependency was unpublished from npm.


The Soviet version does not have this problem. I suggest you reinstall the package on Astra Linux.




It works for me in node.js

Mte90 commented Jul 8, 2016

Tried an apt-get upgrade? maybe it is a problem of dependencies of the system.

arvigeus commented Jul 8, 2016

Found the problem: missing semicolon

Yokai commented Jul 8, 2016

It appears that a directive may have had this effect. +1 for moving to react or jquery rather than angular


This is how it could have been fixed
Armageddon russian cosmonaut fix

RikerW commented Jul 8, 2016

Relavent xkcd.

k-gun commented Jul 9, 2016

I tested it with CodeIgniter and worked well locally. CodeIgniter will keep it warm but I would recommend to use it PHP >= 5.6 with OpCode option on.

murac commented Jul 9, 2016

You guys realize this spaceship launched in 1969, right? It's many years later now. I don't believe this software is still in use today.

RikerW commented Jul 9, 2016 edited

@murac The issue is a joke. And the whole repo is more of historical interest.

piranna commented Jul 9, 2016
markemer commented Jul 9, 2016

@murac The spaceship we are talking about launched April 13, 1970. We're joking about Apollo 13 not 11. Although the fact we have not been back since 1972 makes me unbelievably sad.

paoloo commented Jul 9, 2016

It worked here, have you checked your hardware? I'll attach a schematic to help you in the process.


Well, after reading all the great technical advice here in this thread, i'm reminded once again of that old adage - "sure it's rocket science, but we're all rocket scientists!"


Why are you even trying to stir the tanks? This doesn't sound like something you should do with the CM. Just switch your entire mission to a shuttle design, it's more modern and efficient.

bkix commented Jul 9, 2016

Just wanted you all to know that I'm vegetarian...

RikerW commented Jul 9, 2016

@piranna another relavent xkcd.

piranna commented Jul 9, 2016

@piranna another relavent xkcd.

Isn't there a relevant xkcd strip for everything? ;-)


Could Matt Damon fix this?


This is technically a fatal error. However, due to the heroic efforts of:


and extensive technical support lead by @Gene-Kranz; A crash in production can apparently be avoided.

These are extraordinary people. We can't expect every user to be of this caliber. I suggest this issue be recategorized as mission critical. Even if it is a result of a hardware failure we should improve fault tolerance, and mitigate the impact. Failure is not an option.


Works for me. I guess that you forgot to turn off selinux ;)

fmanno commented Jul 9, 2016

Hi guys. I have a similar problem but it's not on a CSM it's a car and instead of the O2 tanks I have fuel tanks. Any help plz email me the solutio!

mvalipour commented Jul 9, 2016 edited

Works fine on our moon at this end of the galaxy!

laoneo commented Jul 10, 2016

Going to ask Chuck Norris for a hot fix, we will deliver it OTA.

macbury commented Jul 10, 2016

Did you tried to reimplement it in delphi?


Ask the flight attendants or beam me up Scotty.


OMG, it is not wroted in PHP!!!


Downloading more RAM fixes this issue.

Kinaan commented Jul 10, 2016

migrating to ruby and using gem 'apolo-tank-stirrer' fixed it for me. sending a PR shortly.


I tried porting it down to Haskell and it compiles. By the laws of Haskell it should work if it compiles. Verified the same by porting down to Rust too.


I just want it to work..anyone who has a docker image for this?


I think rewriting this to meteorjs will do the trick.


Try running it on the lunar cloud VMs.


Those old lady's forgot to deploy the commit that fixed the bug. I have asked them to re weave it

DJviolin commented Jul 10, 2016 edited

Have you tried Node.js?


I'd rather go for Eclipse luna with Java 8 from , but as it will be dark because of the eclipse we need additional lights.


BTW I have been in a private conference in 1998 where Jim Lovell talked about leadership. Impressive guy.

ksound commented Jul 10, 2016

Successfully reproduced... I see what you mean, but I'm not sure it's actually a bug. Could this be the intended functionality?


I asked the PM, he said QA approved it, so....


This can easily be resolved with a jQuery plugin.


have you tried to turn it off and on again?


try by not using assembly

@Maijin Maijin referenced this issue in radare/radare2 Jul 10, 2016

Add support for AGC - Apollo-11 cpu #5313


I recommend that future astronauts using this software watch this training video in case this happens again in production.

And congrats to @Jim-Lovell, @John-Swigert, @Fred-Haise, @Gene-Kranz, and the many others at NASA and their contractors who managed bring everyone home safely; it was an extraordinary accomplishment.


We should look at other software that NASA has written for solution ideas, such as in HASP for OS/360 on the IBM System/360 running at JSC.


Try deleting the user preference files and rebooting. Someone may have set the cryogenic tanks to 'shaken, not stirred'.

elfan commented Jul 10, 2016

It's a UI problem. The button to stir the tank is too dominant, we should make it a bit smaller and disable it if it is not supposed to be clicked.


sir plz tell me exactly how to do this !!!!!!!!


Try migrating to jQuery 3.0


This is interesting! All the comments here... :D


WARNING: Don't migrate to jQuery 3.0 and try reproducing this issue.

W-M-D commented Jul 11, 2016

confirming @jaysoffian has a working fix.


Thank you for your report. However, the Colossus software was produced for the AGC, which was responsible for either the navigation of the Command Module or the Lunar Module. The problem you've reported can only occur in the Service Module and has nothing to do with AGC functionality.

Please report your bug in the Service Module project.

lfrankel commented Jul 11, 2016 edited

Actually, @MadDonArabian and his team were able to reliably reproduce the cause of the problem.

It's a hardware design defect due to miscommunication. A heating safety system on the the cryo assembly spec from North American originally expected 28v DC. NA asked the subcontractor to change to 65v DC, which the did, but failed to inform their subcontractors in turn. In ground operations, the tank got heated higher than its design limit and the safety system fried itself instead of shutting off when it got 65v pushed through it instead of 28v. The overheating cracked open Teflon insulation on wires inside the tank.

The error experienced by the user is due to the wires causing a short, igniting the Teflon, heating the LOX, and causing it to expand and build pressure, eventually causing the explosion. Wontfix is correct. As stated, @MadDonArabian's team reproduced the circumstances of the ground operations on a spare and got the same result. It's a hardware issue.

(See pp451 - 455 of Apollo by C Murray and C Bly Cox, ISBN 0976000806)


@paoloo - the schematic is for the wrong version. You posted 14. The bug is for 11.

johnidm commented Jul 11, 2016 edited

My solution is to change the astronauts and and re-write in JavaScript

uberTof commented Jul 11, 2016

I can confirm upgrading the hardware to the latest NES Advantage joystick worked for me.

danickgevry commented Jul 11, 2016 edited

Perhaps you might need to put it back to channel 3 ?



Thanks for contacting help desk.

My name is Luke and I am here to help you with your CRYOGENIC TANKS WITH A CIRCUIT FAULT PRESENT ISSUE.

Let's start with what version of CRYOGENIC TANKS WITH A CIRCUIT FAULT PRESENT ISSUE you are using. If version A14, press 1. If greater than A14, press 2.

Great, you selected A14, is that correct? If yes, press 1, if no, press 2.

OK, you have confirmed you are using A14.

Have you checked that the CRYOGENIC TANKS WITH A CIRCUIT FAULT PRESENT ISSUE are charged (green light) and are you using the new lightning charger cables? If not, you can purchase the new cables from our online store and have them shipped to you, just click here if you need them:

Now, if after verifying that the CRYOGENIC TANKS WITH A CIRCUIT FAULT PRESENT ISSUE battery indicator is not green, follow these steps:

  1. Make sure the tanks have power (they are plugged in to a wall outlet).
  2. Make sure the tanks batteries are charged, if not, use a lightning charger to charge them. If you do not have the new lightning charger if you suspect the one you have is faulty you can purchase one from our online store here:
    (Note charging should take 2-3 hours if batteries are depleted.)
  3. Make sure you do not have too many CRYOGENIC TANKS WITH A CIRCUIT FAULT PRESENT ISSUE running at once, if so you will need to close some. Double click the home button and then swipe up to close each open tank.
  4. If this does not resolve the problem you will need to send the CRYOGENIC TANKS WITH A CIRCUIT FAULT PRESENT ISSUE to customer service. Please send me a valid email address and I will forward shipping labels to you. There is no cost to ship the CRYOGENIC TANKS WITH A CIRCUIT FAULT PRESENT ISSUE back to us but you will need to get them to a UPS store or have UPS pick them up from your location.

Please remain on the line for a short survey. This will help us improve our service to you in the future!

tkaefer commented Jul 11, 2016

Anybody called Tom Hanks,Bill Paxton or Kevin Bacon? I've seen a training video, where they were able to fix that.


Your holding it wrong.


Same here


Yea, have you tried Docker?


Its not a bug. It is the limitation of the free version. Kindly upgrade to the premium version to enjoy
hazard free cyro tank stirring.

Just kidding. I know its a bug. But nobody clicks on my "Donate via paypal" button, so I ain't going to fix it.
There.. I said it.

xchehub commented Jul 12, 2016

Please find second source.

bkowald commented Jul 12, 2016

Cannot replicate on my test system. Please zip up all logs and a memory dump and upload it to the support ftp site. We will get back to you within 24 hours. Have a nice day.


It's a network problem. Closing ticket.


This is expected behaviour, In fact its a feature, not a bug.


I was able to fix this by going into the config files and deleting one called "System32".
Apollo is really tough to boot from then on, but it's a solid fix.

Erigne commented Jul 13, 2016

Login to NDS as admin and type "load landing.nlm"

kzarim commented Jul 13, 2016

Sprzedam Opla! :D
But seriously guys, that bug is clearly caused by the team not having daily agile stand-ups! Agile would do that the right way!


I like my O2 shaken not stirred. Do I have to log in a new issue?

rdab100 commented Jul 13, 2016

Sounds like you have run into a firmware bug and need a sensor replaced.... we can arrange for a new one to be fitted next week that should fix this issue so if you can drop when in the area next we can take a look at it.


That's an issue that has been pointed out to us by other users as well.
To get this repaired, please send your unit to our service center. We will repair it/replace it depending on the problem with the pre-requisite being:

  1. you have a valid license of the software; it should not be a pirated copy.
  2. The tank has not been rooted or jailbroken, as it voids the warranty.
@aandis aandis referenced this issue in gratipay/ Jul 14, 2016

Core Radar 67 #704


guys, with this attitude (see almost all comments above), we'll never reach the moon


I just noticed this bug only exists in the 5 bit version. Since we'll be upgrading to the 6 bit version real soon I suggest leaving closed as a WON'T FIX.


I am on Mars for a client assignment right now. So it will be a while before I look into the issue.
Also, I note that you are using a legacy setup that we no longer actively develop.
We will however maintain backward compatibility till 31st Dec 2016. This is part of our strategy to make a streamline move towards cost-effective SpaceX components.
We advice you to upgrade your infrastructure to reflect the same.

That said, we do have a patch for your problem but it wont be until the next release that you can see the fix.
However, if it is very critical for your project, then kindly send over your time signature (in Unix timestamp format) and the value of gravity at your test facility, and one of our engineers will travel back in time to examine and fix the issue.



Is this happening in sandbox, or a production environment? Engineering is saying the described behavior is impossible, it's got to be an instrumentation issue.


Leave it.
until they provide hardware too :)


Your call might be recorded for quality and training purposes


You forgot to import apollo_11.cms...


let me check it out


Dominic A. Imbuga *
Blogs: ||
*Fullstack Developer | Blogger


Skype:Dominic-imbuga_+254 715 180 530 | +254 734 983 442_

_" _Science and technology revolutionize our lives, but memory, tradition
and myth frame our response"

On Mon, Jul 18, 2016 at 9:48 PM, Victor Otieno

You forgot to import apollo_11.cms...

You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
#3 (comment),
or mute the thread



ahmadawais commented Jul 20, 2016 edited

I just solved the issue. Here's how, have a listen!

🔘──────── 00:21


I suggest assigning Stanley Kubrick to fix this issue.

@chrislgarry chrislgarry added the humor label Aug 6, 2016
pantaluna commented Aug 17, 2016 edited

Which version of Apollo are you running? I suggest upgrading from Apollo 11 to Apollo 12

@khash We cannot upgrade simply because by our 60's semver rules the V12 contains "breaking" changes.


This never would have happened when Steve Jobs was alive.

sdhar86 commented Sep 15, 2016

And that's why we should use docker, could have reproduced the issue easily.


works on my machine


You guys are trying to solve an issue which doesn't exist in future.
The Doctor has already solved it in past future.
In his words:
"People assume that time is a strict progression of cause to effect, but actually from a non-linear, non-subjective viewpoint - it's more like a big ball of wibbly wobbly... time-y wimey... stuff."

So, let's stop being geeky and go back to humour, ehh?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment