Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adds Rust deployment capabilities #774

Closed
wants to merge 24 commits into from
Closed

Adds Rust deployment capabilities #774

wants to merge 24 commits into from

Conversation

cflewis
Copy link

@cflewis cflewis commented Jun 17, 2016

THIS CODE DOES NOT WORK YET

This PR contains code that fixes up some of jon-rust-deploy-2 so that it seems to do the right thing, but it hangs waiting for the rust cross-compliation server. The server itself seems to be getting request and returning the response just fine, so there's something here that's waiting and I don't know why. My Javascript is not great, and the asynchroncity here makes things worse for me.

I'm happy to take comments as to what to do next, or just have this PR pulled into the branch and @johnnyman727 continues with it.

@cflewis
Copy link
Author

cflewis commented Jun 17, 2016

My debugging has proved fruitless... I thought it was because post_req.end was never called, but it is, so that's not it.

I do know we reach the L175 block in index.js (Data Complete State) and it starts running the dataFill.
Data is pushed on L113 in usb-connection.js, the transfer starts and then I lose the trail somewhere and the process goes into an infinite loop. I do not know if this is part of the normal state of things.

Theories:

  • Rust compilation server returns garbage and nothing useful is pushed to the Tessel (unlikely).
  • Some sort of closing function needs to run that doesn't, which puts everything into an infinite loop.
  • Some code is missing. My debug statements get to this:
INFO Looking for your Tessel...
DEBUG (discovery) Will scan for LAN devices
DEBUG (discovery) Will scan for USB devices
DEBUG (discovery:usb) Device found.
DEBUG (discovery) Timeout hit! Waiting for pending to finish
DEBUG (discovery) Connection opened: undefined
DEBUG (discovery) Fetching name: undefined
DEBUG (commands:usb) uci,get,system.@system[0].hostname
DEBUG (usb-process) Opening process for: ucigetsystem.@system[0].hostname
DEBUG (usb-process) Death for "ucigetsystem.@system[0].hostname" with exit code: 0
DEBUG (discovery) Done! Stopping.
INFO Connected to belle.
DEBUG (commands:usb) /etc/init.d/tessel-app,stop
DEBUG (usb-process) Opening process for: /etc/init.d/tessel-appstop
DEBUG (usb-process) Death for "/etc/init.d/tessel-appstop" with exit code: 0
DEBUG (commands:usb) rm,-rf,/tmp/remote-script/
DEBUG (usb-process) Opening process for: rm-rf/tmp/remote-script/
DEBUG (usb-process) Death for "rm-rf/tmp/remote-script/" with exit code: 0
DEBUG (commands:usb) mkdir,-p,/tmp/remote-script/
DEBUG (usb-process) Opening process for: mkdir-p/tmp/remote-script/
DEBUG (usb-process) Death for "mkdir-p/tmp/remote-script/" with exit code: 0
DEBUG (commands:usb) tar,-x,-C,/tmp/remote-script/
DEBUG (usb-process) Opening process for: tar-x-C/tmp/remote-script/
INFO Compiling Rust code...

I don't understand why Compiling Rust code... is the last thing I see, the code in the Promise does get executed according to my breakpoints. Perhaps Promises work in a way that I am not sure of.

Regardless, I can't make any progress now. Any help is appreciated.

@rwaldron
Copy link
Contributor

You'll want to run grunt locally and fix any existing issues in the code.

Also, ping @johnnyman727

rwaldron and others added 11 commits June 21, 2016 14:38
… only)

Signed-off-by: Rick Waldron <waldron.rick@gmail.com>
Signed-off-by: Rick Waldron <waldron.rick@gmail.com>
Signed-off-by: Rick Waldron <waldron.rick@gmail.com>
Signed-off-by: Rick Waldron <waldron.rick@gmail.com>
Signed-off-by: Rick Waldron <waldron.rick@gmail.com>
…ms (where unnecessary)

Signed-off-by: Rick Waldron <waldron.rick@gmail.com>
Signed-off-by: Rick Waldron <waldron.rick@gmail.com>
Signed-off-by: Rick Waldron <waldron.rick@gmail.com>
* t2 init: broad cleanup

Signed-off-by: Rick Waldron <waldron.rick@gmail.com>

* t2 init: more tests for --lang=javascript

Signed-off-by: Rick Waldron <waldron.rick@gmail.com>

* t2 init: per review: not necessary to explicitly coerce lang input to string

Signed-off-by: Rick Waldron <waldron.rick@gmail.com>

* t2 init: init.rs.createSampleProgram use fs.copy

Signed-off-by: Rick Waldron <waldron.rick@gmail.com>

* t2 init: special *Error classes are not exported.

Signed-off-by: Rick Waldron <waldron.rick@gmail.com>

* t2 init: reuse rust resource paths

Signed-off-by: Rick Waldron <waldron.rick@gmail.com>

* t2 init: cleanup all resource paths

Signed-off-by: Rick Waldron <waldron.rick@gmail.com>
@cflewis
Copy link
Author

cflewis commented Jun 27, 2016

I've got grunt happy up to testing deploy.js, which is when it hits the hang.

@johnnyman727 will need to take it from here.

Reader({
path: opts.target,
type: 'Directory'
})
.on('error', reject)
.pipe(outgoingPacker)
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are outgoingPacker and post_req the wrong way round here?

@johnnyman727
Copy link
Contributor

@cflewis I had a few hours to dig into the issue tonight. Ultimately, I just patched the cross compilation server to archive the binary on its end before passing back to the CLI which, I think, is the cleaner way to do it. I opened a PR on your branch with the fix but I'm going to keep adding to my PR to make the code more robust (you currently can't control+c to abort once the LEDs start blinking for some reason).

@johnnyman727 johnnyman727 changed the title Improve rust deploy code Adds Rust deployment capabilities Jul 2, 2016
@cflewis
Copy link
Author

cflewis commented Jul 2, 2016

@rwaldron Can you confirm this is rebased as you expect it to be?

@cflewis
Copy link
Author

cflewis commented Jul 3, 2016

@johnnyman727 : I'm not seeing this work, although there may be some sort of user error :/

❯ node ~/Documents/Computing/rust/t2-cli/bin/tessel-2.js run -v --loglevel=debug --rustCC=localhost:8080 .             hello/git/master 
INFO Looking for your Tessel...
DEBUG (discovery) Will scan for LAN devices
DEBUG (discovery) Will scan for USB devices
DEBUG (discovery:usb) Device found.
DEBUG (discovery) Connection opened: undefined
DEBUG (discovery) Fetching name: undefined
DEBUG (commands:usb) uci,get,system.@system[0].hostname
DEBUG (usb-process) Opening process for: ucigetsystem.@system[0].hostname
DEBUG (usb-process) Death for "ucigetsystem.@system[0].hostname" with exit code: 0
DEBUG (discovery) Timeout hit! Waiting for pending to finish
DEBUG (discovery) Done! Stopping.
INFO Connected to belle.
DEBUG (commands:usb) /etc/init.d/tessel-app,stop
DEBUG (usb-process) Opening process for: /etc/init.d/tessel-appstop
DEBUG (usb-process) Death for "/etc/init.d/tessel-appstop" with exit code: 0
DEBUG (commands:usb) rm,-rf,/tmp/remote-script/
DEBUG (usb-process) Opening process for: rm-rf/tmp/remote-script/
DEBUG (usb-process) Death for "rm-rf/tmp/remote-script/" with exit code: 0
DEBUG (commands:usb) mkdir,-p,/tmp/remote-script/
DEBUG (usb-process) Opening process for: mkdir-p/tmp/remote-script/
DEBUG (usb-process) Death for "mkdir-p/tmp/remote-script/" with exit code: 0
Remote run path is /tmp/remote-script/
DEBUG (commands:usb) tar,-x,-C,/tmp/remote-script/
DEBUG (usb-process) Opening process for: tar-x-C/tmp/remote-script/
DEBUG (usb-process) Death for "tar-x-C/tmp/remote-script/" with exit code: 9
ERR! TypeError: Path must be a string. Received undefined

Something isn't right :/

@johnnyman727
Copy link
Contributor

@cflewis I'm not sure why you're hitting that. Here's was the output looks like for me:

➜  rust-demo t2 run . --rustCC=http://192.168.99.100:49160 --usb --loglevel=debug
INFO Looking for your Tessel...
DEBUG (discovery) Will scan for USB devices
DEBUG (discovery:usb) Device found.
DEBUG (discovery) Connection opened: undefined
DEBUG (discovery) Fetching name: undefined
DEBUG (commands:usb) uci,get,system.@system[0].hostname
DEBUG (usb-process) Opening process for: ucigetsystem.@system[0].hostname
DEBUG (usb-process) Death for "ucigetsystem.@system[0].hostname" with exit code: 0
DEBUG (discovery) Timeout hit! Waiting for pending to finish
DEBUG (discovery) Done! Stopping.
INFO Connected to frank.
DEBUG (commands:usb) /etc/init.d/tessel-app,stop
DEBUG (usb-process) Opening process for: /etc/init.d/tessel-appstop
DEBUG (usb-process) Death for "/etc/init.d/tessel-appstop" with exit code: 0
DEBUG (commands:usb) rm,-rf,/tmp/remote-script/
DEBUG (usb-process) Opening process for: rm-rf/tmp/remote-script/
DEBUG (usb-process) Death for "rm-rf/tmp/remote-script/" with exit code: 0
DEBUG (commands:usb) mkdir,-p,/tmp/remote-script/
DEBUG (usb-process) Opening process for: mkdir-p/tmp/remote-script/
DEBUG (usb-process) Death for "mkdir-p/tmp/remote-script/" with exit code: 0
DEBUG (commands:usb) tar,-x,-C,/tmp/remote-script/
DEBUG (usb-process) Opening process for: tar-x-C/tmp/remote-script/
INFO Compiling Rust code remotely...
INFO    Compiling cfg-if v0.1.0
INFO    Compiling libc v0.2.13
INFO    Compiling unix_socket v0.5.0
INFO    Compiling rust_tessel v0.0.1 (https://github.com/tessel/rust-tessel.git#f5a8f9b9)
INFO    Compiling hello v0.0.1 (file:///tmp/tessel-rust-compile11663-541-1384jb6/rust-demo)
INFO
INFO Writing project to RAM on frank (317.44 kB)...
DEBUG (usb-process) Death for "tar-x-C/tmp/remote-script/" with exit code: 0
INFO Deployed.
INFO Running hello...
DEBUG (commands:usb) chmod,+x,/tmp/remote-script/hello
DEBUG (usb-process) Opening process for: chmod+x/tmp/remote-script/hello
DEBUG (usb-process) Death for "chmod+x/tmp/remote-script/hello" with exit code: 0
DEBUG (commands:usb) /tmp/remote-script/hello
DEBUG (usb-process) Opening process for: /tmp/remote-script/hello
I'm blinking! (Press CTRL + C to stop)
^CDEBUG (usb-process) Death for "/tmp/remote-script/hello" with exit code: 9

It looks like your execution didn't even reach the cross compilation server before it failed. It's failing somewhere between running the tar command and the call to compile the bundle remotely.

Are you sure you have a valid Cargo.toml?

Also, when I look at the git history, it appears to be rebased but Github still shows a bunch of commits that are already in master on this PR so :shrugs:.

@johnnyman727
Copy link
Contributor

Why do you have hello/git/master in your command to run? Maybe that's throwing it off.

@cflewis
Copy link
Author

cflewis commented Jul 3, 2016

I'm still seeing the same thing (the /hello/git/master/ is just a shell display thingy) with a fresh clone, using a binary of cargo new hello --bin. What does your TOML look like? Mine is:

[package]
name = "hello"
version = "0.1.0"
authors = ["Chris Lewis <chris@chris.to>"]

[dependencies]

@cflewis
Copy link
Author

cflewis commented Jul 3, 2016

Here's the problem: the path that is being sent to the fs package isn't right:

Error: ENOENT: no such file or directory, stat '/Users/cflewis/Downloads/hello/src/Cargo.toml'

The TOML file is one level up. I'm going to try and see why this string is built wrongly.

@cflewis
Copy link
Author

cflewis commented Jul 3, 2016

Even when I put the TOML there, something barfs in checkConfiguration, but it seems fine. Show me your TOML and I think we will find the problem.

exportables.preBundle = function(opts) {
return new Promise((resolve, reject) => {
// Get details of the project
var details = exportables.meta.checkConfiguration(opts.target);
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is where I get my failure, even if the TOML is copied.

opts.target is set to "/Users/cflewis/Downloads/hello/src", when checkConfiguration is expecting three arguments. When L31 is triggered, the path package barfs as program is undefined. Question is why opts is only one rather than three things, and that I did not have time to work out before I had to stop.

@johnnyman727
Copy link
Contributor

My Cargo.toml is virtually the same:

[package]
name = "hello"
version = "0.0.1"
authors = ["Your Name <you@example.com>"]

[dependencies.rust_tessel]
git="https://github.com/tessel/rust-tessel.git"

I think the issue is actually here where we assume that the Cargo.toml is in the same directory that you pushed from. Is it possible that you are deploying from within YOUR_PROJECT/src instead of YOUR_PROJECT? Perhaps we need to write some logic to traverse up from the pushdir to find the root of the project (ie where the Cargo.toml` lives) just like we do with JS.

@cflewis
Copy link
Author

cflewis commented Jul 4, 2016

No, I'm not doing that :/

Here's the exact commands I'm running:

~/Downloads❯ git clone https://github.com/cflewis/t2-cli.git
Cloning into 't2-cli'...
remote: Counting objects: 5370, done.
remote: Compressing objects: 100% (51/51), done.
remote: Total 5370 (delta 26), reused 0 (delta 0), pack-reused 5319
Receiving objects: 100% (5370/5370), 1.38 MiB | 367.00 KiB/s, done.
Resolving deltas: 100% (3591/3591), done.
Checking connectivity... done.
~/Downloads❯ cargo new hello --bin
~/Downloads❯ docker run -p 49160:8080 c4a0d63dc9cf &
~/Downloads❯ cd t2-cli
❯ git checkout jon-rust-deploy-2                                                                                      t2-cli/git/master 
❯ npm link                                                                                                            t2-cli/git/master 
<SNIP>
❯ cd ../                                                                                                   t2-cli/git/jon-rust-deploy-2 
~/Downloads❯ cd hello
❯ ls                                                                                                                   hello/git/master 
Cargo.toml src
❯ node ../t2-cli/bin/tessel-2.js run -v --rustCC=172.17.0.2:49160 --usb --loglevel=debug .                             hello/git/master 
INFO Looking for your Tessel...
DEBUG (discovery) Will scan for USB devices
DEBUG (discovery:usb) Device found.
DEBUG (discovery) Connection opened: undefined
DEBUG (discovery) Fetching name: undefined
DEBUG (commands:usb) uci,get,system.@system[0].hostname
DEBUG (usb-process) Opening process for: ucigetsystem.@system[0].hostname
DEBUG (usb-process) Death for "ucigetsystem.@system[0].hostname" with exit code: 0
DEBUG (discovery) Timeout hit! Waiting for pending to finish
DEBUG (discovery) Done! Stopping.
INFO Connected to belle.
DEBUG (commands:usb) /etc/init.d/tessel-app,stop
DEBUG (usb-process) Opening process for: /etc/init.d/tessel-appstop
DEBUG (usb-process) Death for "/etc/init.d/tessel-appstop" with exit code: 0
DEBUG (commands:usb) rm,-rf,/tmp/remote-script/
DEBUG (usb-process) Opening process for: rm-rf/tmp/remote-script/
DEBUG (usb-process) Death for "rm-rf/tmp/remote-script/" with exit code: 0
DEBUG (commands:usb) mkdir,-p,/tmp/remote-script/
DEBUG (usb-process) Opening process for: mkdir-p/tmp/remote-script/
DEBUG (usb-process) Death for "mkdir-p/tmp/remote-script/" with exit code: 0
DEBUG (commands:usb) tar,-x,-C,/tmp/remote-script/
DEBUG (usb-process) Opening process for: tar-x-C/tmp/remote-script/
DEBUG (usb-process) Death for "tar-x-C/tmp/remote-script/" with exit code: 9
ERR! TypeError: Path must be a string. Received undefined

@johnnyman727
Copy link
Contributor

johnnyman727 commented Jul 4, 2016

Ah I figured out the issue. You are likely using Node 5.x or Node 6.x which changed the path.basename API to throw an error if the provided path is not a string. The Node 4.x API (which I'm using) just returns undefined if the path is invalid.

The CLI officially only supports Node LTS (4.x for now) but I'll make a quick addition to resolve this. It doesn't matter too much because Rust doesn't use the basename for now.

@johnnyman727
Copy link
Contributor

Although, we don't have a version of the usb dependency that compiles for Node 5.x or Node 6.x so I'm not sure how you would have gotten that version to work at all...

@cflewis
Copy link
Author

cflewis commented Jul 4, 2016

Aha! We will need to make this clearer in the README for Rust types, most of them I have seen use Mac OS X, and if you're using Homebrew, brew install node is going to get you > Node 4. I'll send a PR for that.

I have new weirdness instead :( My compilation server is up, and I can talk to my Tessel, but something isn't communicating correctly:

❯ curl localhost:49160                                                                                      
Tessel Rust Cross Compilation Server!% 
❯ t2 root                                                                                                                 
INFO Looking for your Tessel...
INFO Connected to belle.
INFO Starting SSH Session on Tessel. Type "exit" at the prompt to end.


BusyBox v1.23.2 (2016-04-07 13:52:07 EDT) built-in shell (ash)

Tessel 2  /  Built on OpenWrt
root@belle:~# exit

❯ t2 run -v --rustCC=172.17.0.2:49160 --usb --loglevel=debug .                                                           hello/git/master 
INFO Looking for your Tessel...
DEBUG (discovery) Will scan for USB devices
DEBUG (discovery:usb) Device found.
DEBUG (discovery) Connection opened: undefined
DEBUG (discovery) Fetching name: undefined
DEBUG (commands:usb) uci,get,system.@system[0].hostname
DEBUG (usb-process) Opening process for: ucigetsystem.@system[0].hostname
DEBUG (usb-process) Death for "ucigetsystem.@system[0].hostname" with exit code: 0
DEBUG (discovery) Timeout hit! Waiting for pending to finish
DEBUG (discovery) Done! Stopping.
INFO Connected to belle.
DEBUG (commands:usb) /etc/init.d/tessel-app,stop
DEBUG (usb-process) Opening process for: /etc/init.d/tessel-appstop
DEBUG (usb-process) Death for "/etc/init.d/tessel-appstop" with exit code: 0
DEBUG (commands:usb) rm,-rf,/tmp/remote-script/
DEBUG (usb-process) Opening process for: rm-rf/tmp/remote-script/
DEBUG (usb-process) Death for "rm-rf/tmp/remote-script/" with exit code: 0
DEBUG (commands:usb) mkdir,-p,/tmp/remote-script/
DEBUG (usb-process) Opening process for: mkdir-p/tmp/remote-script/
DEBUG (usb-process) Death for "mkdir-p/tmp/remote-script/" with exit code: 0
DEBUG (commands:usb) tar,-x,-C,/tmp/remote-script/
DEBUG (usb-process) Opening process for: tar-x-C/tmp/remote-script/
INFO Compiling Rust code remotely...
ERR! Detected CLI crash { [Error: connect EHOSTUNREACH 0.0.192.8:80 - Local (10.0.1.5:50283)]
ERR!   code: 'EHOSTUNREACH',
ERR!   errno: 'EHOSTUNREACH',
ERR!   syscall: 'connect',
ERR!   address: '0.0.192.8',
ERR!   port: 80 } Error: connect EHOSTUNREACH 0.0.192.8:80 - Local (10.0.1.5:50283)
ERR!     at Object.exports._errnoException (util.js:873:11)
ERR!     at exports._exceptionWithHostPort (util.js:896:20)
ERR!     at connect (net.js:865:16)
ERR!     at net.js:1000:7
ERR!     at GetAddrInfoReqWrap.asyncCallback [as callback] (dns.js:64:16)
ERR!     at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:83:10)
ERR! Detected CLI crash { [Error: socket hang up] code: 'ECONNRESET' } Error: socket hang up
ERR!     at createHangUpError (_http_client.js:211:15)
ERR!     at Socket.socketCloseListener (_http_client.js:243:23)
ERR!     at emitOne (events.js:82:20)
ERR!     at Socket.emit (events.js:169:7)
ERR!     at TCP._onclose (net.js:486:12)
INFO Crash Reported: http://crash-reporter.tessel.io/crashes?fingerprint=0x3edeaecb
INFO Crash Reported: http://crash-reporter.tessel.io/crashes?fingerprint=0x1c0ea9e4
^CDEBUG (usb-process) Death for "tar-x-C/tmp/remote-script/" with exit code: 9

0.0.192.8:80 makes me think that's some internal representation for chatting over USB, but I am not sure why my Tessel would be hanging up.

@johnnyman727
Copy link
Contributor

Try adding "http://" in front of the IP address.

@cflewis
Copy link
Author

cflewis commented Jul 4, 2016

Awesome, adding HTTP does indeed make this work. I'll add a patch that'll force http:// if not provided.

@cflewis
Copy link
Author

cflewis commented Jul 4, 2016

I'm happy for this to land into master now, pending the CI checks being made happy (which I do not think I have permission to see as I don't have write permission to t2-cli). I think I've blundered into most of the edge cases :)

@@ -86,6 +86,10 @@ exportables.remoteRustCompilation = (opts) => {
var buffers = [];

// Parse out the components of the remote rust compilation server
if (!opts.rustCC.startsWith('http://')) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You may want to also check that it starts with https:// in case someone deploys their own cross compilation server somewhere behind SSL (or if we do that...).

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

@johnnyman727
Copy link
Contributor

@cflewis after addressing my two comments, can you re-fetch from this endpoint and then git rebase origin/master again? After I did that on my own test branch, all the commits that were already in master didn't show up and only 11 files were changed.

@rwaldron has requested that we merge #787 before this one. It cleans up some of the deploy process but shouldn't affect our work with this PR.

After that, we'll need one more review before we can merge. @rwaldron are you up for that?

@johnnyman727
Copy link
Contributor

And, as discussed in this issue, add yourself to the contributor list!

@johnnyman727
Copy link
Contributor

For convenience:

  • Fix grunt lint warning about semicolon
  • Update rustCC protocol check to include https
  • Add @cflewis to the contributors field of the package.json
  • Get a review from one more Team Member

@cflewis
Copy link
Author

cflewis commented Jul 5, 2016

  • Fix grunt lint warning about semicolon
  • Update rustCC protocol check to include https
  • Add @cflewis to the contributors field of the package.json
  • Get a review from one more Team Member

I must be doing something wrong here with the rebasing (yay Git).

❯ git fetch upstream master                                                                                
From https://github.com/tessel/t2-cli
 * branch            master     -> FETCH_HEAD
❯ git rebase upstream/master
Current branch jon-rust-deploy-2 is up to date.

and GitHub still shows the commits in this PR :(

Do you want to squash this PR into one commit? That might get us the right UI display at least.

@johnnyman727
Copy link
Contributor

@cflewis I'm not sure why the rebase isn't working (I'm wondering if it's also why Appveyer is upset). Want to just close this one in favor of #790?

@cflewis
Copy link
Author

cflewis commented Jul 6, 2016

Yeah, sounds good.

@cflewis cflewis closed this Jul 6, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants