New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Outdated docs for hacking on sandstorm shell #3047

Open
dmitry-sher opened this Issue Feb 27, 2018 · 21 comments

Comments

Projects
None yet
6 participants
@dmitry-sher
Copy link

dmitry-sher commented Feb 27, 2018

UPDATE

Easiest way to hack on sandstorm shell is as follows:

  • install latest VirtualBox (5.2.6 is good)
  • import VM, made by @xet7 . read #3036 for details. VM link
  • set up VM's networking as appropriate (i prefer NAT with some port like host's 4222 as guest 22 and host's 127.0.0.1:6080 -> guest's 0.0.0.0:6080). ssh to VM with wekan:wekan ssh -p 4222 wekan@localhost. You can add your keys ssh-copy-id -p 4222 wekan@localhost
  • update /opt/sandstorm/sandstorm.conf in VM. set BIND_IP=0.0.0.0. comment out #UPDATE_CHANNEL=dev. restart sandstorm sudo service sandstorm restart. Now you can access sandstorm from host OS on port 6080
  • add wekan user to sandstorm group. sudo usermod -a -G sandstorm wekan. logout and login back for changes to take place.
  • go to ~/repos/sandstorm and make update
  • run sandstorm dev-shell from the repo root.

Below are my previous attempts to do it on Ubuntu 16.04.3. You can safely skip it.

  1. The commands
cd shell
sudo service sandstorm stop-fe
./run-dev.sh

Are not working anymore. Current command is sandstorm dev-shell in project root.

If you run it, you would see a multitude of problems.

1a. Preparations to run hacking.

cd shell
meteor npm install
  1. When trying to run sandstorm dev-shell. From the latest master commit 760faa2
=> Errors prevented startup:                  
   
   While processing files with fourseven:scss (for target web.browser):
   /client/styles/shell.scss: Scss compiler error: File to import:
   {}/client/styles/_icons.scss not found in file:
   /home/sherd/sandstorm/shell/{}/client/styles/shell.scss

As you can see, the file is missing in the sources:
https://github.com/sandstorm-io/sandstorm/tree/master/shell/client/styles.
It's generated. See how to generate it below.

  1. You would miss capnp.node and capnp.js. You can find them in /opt/sandstorm/latest/node_modules. Copy everything from that folder to shell/node_modules.

  2. There are some missing capnp files, like payments.capnp. You can find all current capnp files in project in src/sandstorm/. Copy them to shell/node_modules/sandstorm.

@dmitry-sher

This comment has been minimized.

Copy link
Author

dmitry-sher commented Feb 27, 2018

ok nice... the steps to get that _icons.scss:

  1. go to icons in project root
  2. run npm install
  3. ensure you have grunt: npm install -g grunt
  4. run grunt

@dmitry-sher dmitry-sher changed the title missing _icons.scss Outdated docs for hacking on sandstorm shell Feb 27, 2018

@dmitry-sher

This comment has been minimized.

Copy link
Author

dmitry-sher commented Feb 27, 2018

Problem is much bigger than that.
That whole part of the docs about installing from source is outdated.
Requirements changed a lot.
I will write down what i had to do in order to get it running, if i will succeed.

Before running make or any other steps from Option 4: Installing from Source, do these (all commands are for Ubuntu 16.04):

  1. Install cmake: sudo apt-get install cmake -y. Install clang-6.0:
wget -O - https://apt.llvm.org/llvm-snapshot.gpg.key | sudo apt-key add -
sudo apt-add-repository "deb http://apt.llvm.org/xenial/ llvm-toolchain-xenial-6.0 main"
sudo apt-get update
sudo apt-get install -y clang-6.0
  1. Install Go.
sudo add-apt-repository ppa:gophers/archive
sudo apt-get update
sudo apt-get install golang-1.9-go

Add /usr/lib/go-1.9/bin to PATH.

@dmitry-sher

This comment has been minimized.

Copy link
Author

dmitry-sher commented Feb 27, 2018

Right now i cannot make boringssl because it wants clang-6.0 and the clang-6.0 doesn't want to install on my ubuntu 16.04 because...

E: Failed to fetch http://apt.llvm.org/xenial/dists/llvm-toolchain-xenial-6.0/main/binary-amd64/Packages.gz  Hash Sum mismatch

update: here is relevant bug from llvm bugzilla. it seems closed.
https://bugs.llvm.org/show_bug.cgi?id=36396

update 2: looks it can be a CDN problem. another run and apt-get update worked.

@xet7

This comment has been minimized.

Copy link
Contributor

xet7 commented Feb 27, 2018

@dmitry-sher

I made Debian 9 VirtualBox Appliance where Wekan and Sandstorm compile successfully:
https://github.com/wekan/wekan/releases

There is Wekan-Sandstorm.ova.

@dmitry-sher

This comment has been minimized.

Copy link
Author

dmitry-sher commented Feb 27, 2018

Ok, now the sandstorm is not compiling. Not that I think it's really necessary. I just thought that maybe, if you want to hack on shell, you need to get it make-able first. I will try to continue.

✘ compile: sandstorm/gateway.c++
    /ekam-provider/canonical/sandstorm/gateway.c++:364:1: warning: control may 
      reach end of non-void function [-Wreturn-type]
    }
    ^
    /ekam-provider/canonical/sandstorm/gateway.c++:831:42: error: no member 
      named 'try_emplace' in 'std::map<kj::StringPtr, sandstorm::GatewayService:
      :ForeignHostnameEntry, std::less<kj::StringPtr>, std::allocator<std::pair<
      const kj::StringPtr, sandstorm::GatewayService::ForeignHostnameEntry> > >'
        auto insertResult = foreignHostnames.try_emplace(key, kj::mv(entry));

update try_emplace is c++17 thing. Maybe something is wrong with my compiler..

update 2 i've updated gcc to 7.2.0:

sudo apt-get install build-essential software-properties-common -y && \
sudo add-apt-repository ppa:ubuntu-toolchain-r/test -y && \
sudo apt-get update -y && \
sudo apt-get install gcc-7 g++-7 -y && \
sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-7 60 --slave /usr/bin/g++ g++ /usr/bin/g++-7 && \
sudo update-alternatives --config gcc

now we have different error.

@dmitry-sher

This comment has been minimized.

Copy link
Author

dmitry-sher commented Feb 27, 2018

When I try to run make in sandstorm project root with gcc-7.2.0 I get this error:

✘ compile: sandstorm/sandstorm-http-bridge.c++
    In file included from /ekam-provider/canonical/sandstorm/sandstorm-http-
      bridge.c++:36:
    In file included from /usr/bin/../lib/gcc/x86_64-linux-gnu/7.2.0/../../../..
      /include/c++/7.2.0/map:60:
    In file included from /usr/bin/../lib/gcc/x86_64-linux-gnu/7.2.0/../../../..
      /include/c++/7.2.0/bits/stl_tree.h:63:
    In file included from /usr/bin/../lib/gcc/x86_64-linux-gnu/7.2.0/../../../..
      /include/c++/7.2.0/bits/stl_algobase.h:67:
    In file included from /usr/bin/../lib/gcc/x86_64-linux-gnu/7.2.0/../../../..
      /include/c++/7.2.0/bits/stl_iterator.h:66:
    /usr/bin/../lib/gcc/x86_64-linux-gnu/7.2.0/../../../../include/c++/7.2.0/
      bits/ptr_traits.h:133:9: error: 'rebind' declared as a pointer to a 
      reference of type 'sandstorm::SessionContext::Client &'
            using rebind = _Up*;

update the error is meaningless. It should be smth with this part of sandstorm-http-bridge.c++:

std::map<kj::StringPtr, SessionContext::Client&> sessions;

see that & there? is that wrong? i'm not a c++ ninja for almost 15 years now...

@dmitry-sher

This comment has been minimized.

Copy link
Author

dmitry-sher commented Feb 27, 2018

@xet7 i downloaded Wekan-Sandstorm-0.77.ova - 4.1 GB . For Developers compiling Wekan, Sandstorm, Wekan for Sandstorm, etc. two times and each time when i try to import it into my VirtualBox 5.1.20 r114629 (Qt5.6.2) on MacOS Sierra 10.12.6 (16G29) it fails with following error:

Error reading OVA '/Users/dmitrysher/Downloads/chrome/Wekan-Sandstorm-0.77.ova' (VERR_TAR_UNEXPECTED_EOS).

File size is 4398810112 bytes, sha1 hash ba1f92b2577a32bce7ee4a42b696a0a7f7191599 made with command

$ shasum -v
5.84
$ shasum /Users/dmitrysher/Downloads/chrome/Wekan-Sandstorm-0.77.ova
ba1f92b2577a32bce7ee4a42b696a0a7f7191599
@dmitry-sher

This comment has been minimized.

Copy link
Author

dmitry-sher commented Feb 27, 2018

What a nice catch-22.

~/sandstorm/shell$ sandstorm dev-shell
sandstorm dev-shell: please run this from the root of your Sandstorm source tree
Try 'sandstorm dev-shell --help' for more information.
sherd@osboxes:~/sandstorm/shell$ cd ..
sherd@osboxes:~/sandstorm$ sandstorm dev-shell --help
Usage: sandstorm dev-shell [<option>...] [<meteor-arg>...]

Runs the Sandstorm shell in development mode. For use in developing Sandstorm
itself. Must be run from the `shell` subdirectory of the Sandstorm source code.

Do you see that?
If you run the command from project root, it starts to run but eventually fails. And help message asks to run the command from shell subdir.
But when you run it from shell subdir, it asks to run command from project root.

@dmitry-sher

This comment has been minimized.

Copy link
Author

dmitry-sher commented Feb 27, 2018

I think i will give up for today. It's already evening in moscow. Short summary of the day:

  • docs on site are obsolete
  • source docs are incomplete
  • source docs and help messages are contradictory
  • VM from @xet7 is not working
  • new dev-shell command when i run it from project root is not working at my VM due to multitude of problems, the last problem is message kj/filesystem-disk-unix.c++:1668: warning: PWD environment variable doesn't match current directory; pwd = /home/sherd/sandstorm. It should be somehow connected to what dir it is running from.. somehow..
  • i was ready to dive into c++ code and fix or debug those problems, but i couldn't build the sandstorm c++ code because of some c++17 magic that i don't understand.
@xet7

This comment has been minimized.

Copy link
Contributor

xet7 commented Feb 27, 2018

@dmitry-sher

Your VirtualBox version 5.1.20 r114629 (Qt5.6.2) for Mac OS X is really old.

That VM has been done yesterday with newest VirtualBox 5.2.6, you can
download it from https://www.virtualbox.org , it is compatible with that VM.

You can also check SHA256SUMS.txt that your download is not corrupted:
https://releases.wekan.team/

For me downloading that VM, importing to VirtualBox 5.2.6 and starting it works fine.

Current install Sandstorm from source instructions are at:
https://docs.sandstorm.io/en/latest/install/#option-4-installing-from-source

My previous issue about compiling sandstorm is #3036 , dependencies are currently these. On Debian 9 clang-6.0 is at Debian package repos.

Currently I have these capnproto packages installed, you may need some of those maybe, if those are not installed with with that sudo apt line.

ii  capnproto                             0.6.1-1                             amd64        tool for working with the Cap'n Proto data interchange format
ii  libcap-dev:amd64                      1:2.25-1.2                          amd64        POSIX 1003.1e capabilities (development)
ii  libcap-ng0:amd64                      0.7.7-3.1+b1                        amd64        An alternate POSIX capabilities library
ii  libcap2:amd64                         1:2.25-1.2                          amd64        POSIX 1003.1e capabilities (library)
ii  libcap2-bin                           1:2.25-1.2                          amd64        POSIX 1003.1e capabilities (utilities)
ii  libcapnp-0.6.1                        0.6.1-1                             amd64        Cap'n Proto C++ library
ii  libcapnp-dev                          0.6.1-1                             amd64        Cap'n Proto C++ library (development files)

Also because Debian did not have sudo installed, I installed it:

su
(password here)
apt install sudo
exit

Then continue installing:

sudo apt install build-essential libcap-dev xz-utils zip \
unzip strace curl clang clang-6.0 discount git python zlib1g-dev golang \
cmake autoconf

mkdir ~/go

mkdir ~/repos
cd ~/repos
git clone https://github.com/sandstorm-io/sandstorm.git
cd sandstorm
make

For that VirtualBox VM, I did install Sandstorm normally, without compiling it from source, from https://sandstorm.io/install . I did install it in dev mode. I did also compile sandstorm from source on that VM. To prevent Sandstorm from auto-updating, dev channel needs to be commented out at /opt/sandstorm/sandstorm.conf . Then install compiled sandstorm version, more info at:
https://docs.sandstorm.io/en/latest/install/#option-4-installing-from-source

npm for Debian is possible to install from here:
https://nodejs.org/en/download/package-manager/

@kentonv uses Debian for developing Sandstorm.

@dmitry-sher

This comment has been minimized.

Copy link
Author

dmitry-sher commented Feb 28, 2018

Thnx @xet7 , upgrading VirtualBox helped. My bad.

@dmitry-sher

This comment has been minimized.

Copy link
Author

dmitry-sher commented Feb 28, 2018

Anyway, the problem with hacking the shell is the same:

  • running sandstorm dev-shell from repo root brings the warning kj/filesystem-disk-unix.c++:1668: warning: PWD environment variable doesn't match current directory; pwd = /home/wekan/repos/sandstorm and some broken pipe error
  • running sandstorm dev-shell from the shell subfolder brings error sandstorm dev-shell: please run this from the root of your Sandstorm source tree

Full log

wekan@wekan-sandstorm:~/repos/sandstorm$ sandstorm dev-shell
[[[[[ ~/repos/sandstorm/shell ]]]]]           

=> Started proxy.                             
=> Meteor 1.6.1 is available. Update this project with 'meteor update'.
W20180228-09:20:49.375(2)? (STDERR) kj/filesystem-disk-unix.c++:1668: warning: PWD environment variable doesn't match current directory; pwd = /home/wekan/repos/sandstorm
W20180228-09:20:49.938(2)? (STDERR) Unhandled exception:  { Error: sendmsg(fd, &msg, 0): Broken pipe
W20180228-09:20:49.939(2)? (STDERR)     at Object.exports.connect (/home/wekan/repos/sandstorm/node_modules/capnp.js:265:35)
W20180228-09:20:49.940(2)? (STDERR)     at core.js (server/core.js:662:40)
W20180228-09:20:49.940(2)? (STDERR)     at fileEvaluate (packages/modules-runtime.js:343:9)
W20180228-09:20:49.941(2)? (STDERR)     at require (packages/modules-runtime.js:238:16)
W20180228-09:20:49.942(2)? (STDERR)     at /home/wekan/repos/sandstorm/shell/.meteor/local/build/programs/server/app/app.js:17299:1
W20180228-09:20:49.943(2)? (STDERR)     at infos.forEach.info (/home/wekan/repos/sandstorm/shell/.meteor/local/build/programs/server/boot.js:414:13)
W20180228-09:20:49.944(2)? (STDERR)     at Array.forEach (<anonymous>)
W20180228-09:20:49.944(2)? (STDERR)     at /home/wekan/repos/sandstorm/shell/.meteor/local/build/programs/server/boot.js:413:9
W20180228-09:20:49.945(2)? (STDERR)     at /home/wekan/repos/sandstorm/shell/.meteor/local/build/programs/server/boot.js:463:5
W20180228-09:20:49.945(2)? (STDERR)     at Function.run (/home/wekan/repos/sandstorm/shell/.meteor/local/build/programs/server/profile.js:510:12)
W20180228-09:20:49.946(2)? (STDERR)     at /home/wekan/repos/sandstorm/shell/.meteor/local/build/programs/server/boot.js:462:11
W20180228-09:20:49.946(2)? (STDERR)   cppFile: 'node-capnp/capnp.cc',
W20180228-09:20:49.947(2)? (STDERR)   line: 2062,
W20180228-09:20:49.948(2)? (STDERR)   kjType: 'disconnected' }
@dmitry-sher

This comment has been minimized.

Copy link
Author

dmitry-sher commented Feb 28, 2018

After i made make update, sandstorm dev-shell worked from the project root.

@urakagi

This comment has been minimized.

Copy link
Contributor

urakagi commented Mar 6, 2018

Hi @dmitry-sher, I'm also facing sandstorm-http-bridge.c++: std::map<kj::StringPtr, SessionContext::Client&> sessions; problem. How did you solve this? I have trouble about localization but I'm not able to build sandstorm now.

@dmitry-sher

This comment has been minimized.

Copy link
Author

dmitry-sher commented Mar 6, 2018

@urakagi I gave up and used pre-built VM. Read first comment: #3047 (comment) . BTW, what OS are you using?

I have seen a difference between my initial VM and xet7's one: I had Ubuntu 16.03 with gcc 7.2.0. xet7's: Debian 9 with gcc 7.3.0

@urakagi

This comment has been minimized.

Copy link
Contributor

urakagi commented Mar 7, 2018

I'm using Ubuntu 16.04. I was able to build up hmm.

@kentonv

This comment has been minimized.

Copy link
Member

kentonv commented Mar 11, 2018

Hi all,

I eliminated the Clang dependency because it trips people up. The build now downloads Clang binaries directly from the Chromium project, so you don't have to install it from apt.llvm.org or whatever. See commit 4463c3f.

I've verified that with this change, the build works correctly in a fresh Debian Stretch VM.

@Michael-S

This comment has been minimized.

Copy link
Contributor

Michael-S commented Mar 11, 2018

Very nice, thank you. Separately, I have a list of Sandstorm dependencies other than Clang for Fedora 27. Do you think it's worth adding that to docs/install.md underneath the Debian/Ubuntu section?

@ocdtrekkie

This comment has been minimized.

Copy link
Contributor

ocdtrekkie commented Mar 16, 2018

@Michael-S It's probably going to be useful to someone, probably wouldn't hurt to have it there, and it may end up helping people with other distros too.

@kentonv

This comment has been minimized.

Copy link
Member

kentonv commented Mar 17, 2018

@Michael-S Yes, that would be great.

@Michael-S

This comment has been minimized.

Copy link
Contributor

Michael-S commented Mar 17, 2018

Okay, done with #3060

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment