test-mininodes-ubuntu1604-arm64_odroid_c2-2 needs a manual e2fsck

[Trott]

File system is in read-only mode. Probably needs a manual e2fsck (per @rvagg). I can't even log into the thing (and neither can @refack) so I'm afraid someone else will have to take care of that.

@nodejs/build

One of multiple recent failures (no successes recently):

https://ci.nodejs.org/job/node-test-commit-arm/12017/nodes=ubuntu1604-arm64_odroid_c2/console

Started by upstream project "node-test-commit-arm" build number 12017
originally caused by:
 Started by upstream project "node-test-commit" build number 12304
 originally caused by:
  Started by upstream project "node-test-pull-request" build number 10037
  originally caused by:
   Started by user Jason Ginchereau
[EnvInject] - Loading node environment variables.
Building remotely on test-mininodes-ubuntu1604-arm64_odroid_c2-2 (ubuntu1604-arm64_odroid_c2) in workspace /home/iojs/build/workspace/node-test-commit-arm/nodes/ubuntu1604-arm64_odroid_c2
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/nodejs/node.git # timeout=10
ERROR: Error fetching remote repo 'origin'
hudson.plugins.git.GitException: Failed to fetch from https://github.com/nodejs/node.git
	at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:817)
	at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1084)
	at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1115)
	at hudson.scm.SCM.checkout(SCM.java:496)
	at hudson.model.AbstractProject.checkout(AbstractProject.java:1281)
	at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:604)
	at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:529)
	at hudson.model.Run.execute(Run.java:1728)
	at hudson.matrix.MatrixRun.run(MatrixRun.java:146)
	at hudson.model.ResourceController.execute(ResourceController.java:98)
	at hudson.model.Executor.run(Executor.java:405)
Caused by: hudson.plugins.git.GitException: Command "git config remote.origin.url https://github.com/nodejs/node.git" returned status code 255:
stdout: 
stderr: error: could not lock config file .git/config: Read-only file system

	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1924)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1892)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1888)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1533)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1545)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.setRemoteUrl(CliGitAPIImpl.java:1179)
	at hudson.plugins.git.GitAPI.setRemoteUrl(GitAPI.java:160)
	at sun.reflect.GeneratedMethodAccessor52.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:895)
	at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:870)
	at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:829)
	at hudson.remoting.UserRequest.perform(UserRequest.java:153)
	at hudson.remoting.UserRequest.perform(UserRequest.java:50)
	at hudson.remoting.Request$2.run(Request.java:336)
	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at hudson.remoting.Engine$1$1.run(Engine.java:94)
	at java.lang.Thread.run(Thread.java:748)
	at ......remote call to JNLP4-connect connection from wsip-70-167-220-148.ph.ph.cox.net/70.167.220.148:36794(Native Method)
	at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1545)
	at hudson.remoting.UserResponse.retrieve(UserRequest.java:253)
	at hudson.remoting.Channel.call(Channel.java:830)
	at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:257)
	at com.sun.proxy.$Proxy67.setRemoteUrl(Unknown Source)
	at org.jenkinsci.plugins.gitclient.RemoteGitImpl.setRemoteUrl(RemoteGitImpl.java:295)
	at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:805)
	... 11 more
ERROR: Error fetching remote repo 'origin'
Run condition [Always] enabling perform for step [[]]
TAP Reports Processing: START
Looking for TAP results report in workspace using pattern: *.tap
Did not find any matching files. Setting build result to FAILURE.
Checking ^not ok
Jenkins Text Finder: File set '*.tap' is empty
Notifying upstream projects of job completion
Finished: FAILURE

[rvagg]

Sorry, the reason you couldn't log in is probably because we're now having to use user odroid to log in rather than root, inventory.cfg has now been updated to reflect this.

So in this situation, where you get "Read-only file system", you can reboot and cross your fingers that the file system check that's done during boot to rectify this does a successful job. Or, you can do it yourself to make sure since you can still log in.

This is what I've just done:

• Log in (using odroid)
• touch foo to confirm that the file system is in "Read-only"
• Run sudo e2fsck -y /dev/mmcblk0p2 (root device confirmed using df)

This is essentially what's done on boot with a dirty filesystem, by running it yourself you can catch any problems that may cause a boot to fail and prompt for what to do on the console—which we can't do and would rather avoid. When you have a file system in read-only then it's safe (mostly) to run this and make fixes.

The machine is running again. I've logged back in and done a sudo rm -rf /home/iojs/build/workspace just to start from a fresh workspace cause it may have been compromised during whatever the failure was that happened. It's been brought back online now to.