Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

separate valiation and setup steps from install process #195

Closed
wants to merge 22 commits into from

3 participants

@mpapis

still work in progress, opened PR so the team can check on the commits and let me know if there is something wrong or not acceptable.

@timfel
Owner

looks good to me. does "work in progress" mean you'll refactor this some more?

@mpapis

yes need more work on ./update.sh so it can be used by rvm, basically want to separate download from installation and allow installing to different directory then the checkout dir

@mpapis

It's not tested yet, please somebody review it if it's going in the right direction before I spent time testing if it works (although only minor adjustments might be needed if any)

@Monty
Collaborator

AFAIK there is no ${MAGLEV_SOURCE}/rake, and ${MAGLEV_SOURCE}/bin/rake can't be used to start MagLev or do many operations that affect MagLev infrastructure even if the server is already running.

ok, will revert to which rake check, but is it not a total fail if there is no rake to bootstrap?

@Monty
Collaborator

GemStone an be installed anywhere, not just the MagLev parent directory. For dev, the Maglev parent directory is convenient, especially if you;re running several versions. In production GemStone may be somewhere in /opt/gemstone/

yes, but this scenario was not supported by the installer, so it's a good addition?

@Monty
Collaborator

I can see why you might want $MAGLEV_SOURCE to be different from $MAGLEV_HOME, however there are shell scripts that use the link in $MAGLEV_HOME directly

monty: $ grep HOME $MAGLEV_HOME/bin/maglev | grep gemstone
[ -z "$GEMSTONE" ] && export GEMSTONE=$MAGLEV_HOME/gemstone
monty: $ grep HOME $MAGLEV_HOME/bin/maglev-ruby | grep gemstone
-G Use /opt/gemstone instead of MAGLEV_HOME for locks and logs
export GEMSTONE=$MAGLEV_HOME/gemstone
# echo "Setting \$GEMSTONE to \$MAGLEV_HOME/gemstone"

Collaborator

When you run maglev-ruby or rvm use magev; ruby, it's dependent on the gemstone server having already been started. Since it's a database it may be helpful to think 'postgres' rather than the JVM.

All Ruby source code is loaded into the database server and compiled/executed from there. It's just that MagLev provides a full Ruby parser/jit-compiler as its database language. Running ruby causes the database to spawn a (MagLev) ruby VM.

so the copy to target (3aa475c#L0R232) should happen before any setup steps (3aa475c#L0R168) ?

As for the database thinking, I understand when the script is ran by users the it's in their best interest to already start, but if the script is ran by automated means then it's not the necessary step, the start will be delayed to a later time when MagLev is used or copied to target system.

Collaborator

I'll comment some more in the morning. I'm on East Coast time at my in-laws until the 19th.

Collaborator

IMHO, there is a better way to keep out of the kind of trouble Charles hit. Rid MagLev of any dependencies on any other Rubies.
e.g. use '$MAGLEV_HOME/bin/maglev start' instead of 'rake maglev:start'.

Those rake dependencies are only there because it's easier to write rake scripts than shell scripts. I've never liked that.

I'm certain that $MAGLEV_HOME/bin/maglev could be enhanced to do all MagLev infrastructure tasks and never use rake except for user tasks. It already starts/stops/restarts GemStone, there's no reason it couldn't also create/destroy stones and take snapshots.

I might have to check what code 'rake build:maglev' runs, but all the other tasks currently done by rake in update.sh would be trivial in a shell script.

if it's just a runner then yeah extracting it is the best way, I will have a look if I can help with it

Collaborator

I took a quick hack in the branch monty/remove_rake_from_admin_commands

'bin/maglev start' supposedly checks that prims are loaded and loads them if they aren't.

116 ensurePrimsLoaded() {
117   $TOPAZ_CMD << EOF > /dev/null
118     set gemstone $STONENAME
119     login
120     obj RubyPrimsLoaded
121     quit
122 EOF
123   result=$?
124   if [ "$result" -ne 0 ] ; then
125     echo "Loading kernel. It may take a few seconds..."
126     logfile="$MAGLEV_HOME/log/maglev/allprims.log"
127     $TOPAZ_CMD << EOF > $logfile
128       set gemstone $STONENAME
129       iferr 1 exit 3
130       input $MAGLEV_HOME/src/smalltalk/ruby/allprims.topaz
131       exit 0
132 EOF
133     result=$?
134     if [ "$result" -ne 0 ] ; then
135         echo "FAILED LOADING PRIMS: $result See $logfile"
136     fi
137   fi
138 }

A while back the prim loading process changed. Apparently 'rake build:maglev' was modified to reflect that but 'bin/maglev start' was not.
That makes things more of a PITA. It may have to wait until I get back from vacation where I have my usual workstation to figure out
how to modify 'bin/maglev' so that it mirrors 'rake build:maglev'.

Here's what I see running my modified 'update.sh'

+ echo '[Info] Starting MagLev stone (loading kernel classes)'
[Info] Starting MagLev stone (loading kernel classes)
+ bin/maglev start
startstone[Info]: Starting Stone repository monitor 'maglev'.
startstone[Info]: GemStone server 'maglev' has been started, process 11814 .
Loading kernel. It may take a few seconds...
FAILED LOADING PRIMS: 1 See /Users/monty/MagLev/tmp/maglev-norake/log/maglev/allprims.log

The rakefiles were contributed by Otto Behrens and adapted by Peter McLain. They are overkill for most newbies, but allow creating/managing multiple stones on the same machine. They'll remain, but I could change update.sh to only handle teh default stone name of 'maglev', not do backups, etc.

$ rake -T stone
(in /Users/monty/MagLev/tmp/maglev-norake)
rake stone:all[task_name]        # Invoke a task on all MagLev servers
rake stone:create[server_name]   # Create a new MagLev server and repository
rake stone:destroy[server_name]  # Destroy an existing MagLev server and repository
rake stone:list                  # List MagLev servers managed by this Rakefile
Collaborator

Did a quick proof of concept to make the "maglev start" shell script load prime without using rake. Almost no error checking, needs refactoring/cleanup, but at least it works.

git co monty/remove_rake_from_admin_commands
bash -x update.sh

Once you see "Loading kernel." you can tail -f log/maglev/loadprims.log in another window to see whats actually going on.

Collaborator

POC aside, I'm not sure it's worth the effort to replace the magLev admin rake scripts with enhancements to the bin/maglev shell script. The rake scripts have been tested/used for a long time, and the only benefit is you don't have to have a system ruby/rake in your path in order to install MagLev.

I think cleaning up/refactoring the installation process is worthwhile. Just not getting rid of having ruby/rake as a prerequisite.

I already get past the topaz invocation, for now only parts I have not touched is ERB for the file above and rdoc - for the second I think maglev is already set up and it's ruby + rake could be used

Collaborator

That sounds correct. I may not have time to check before I get back to Portland Tuesday evening -- I'm on the road today.

@mpapis mpapis referenced this pull request in rvm/rvm
Closed

make use of maglev install script #1382

@mpapis

@Monty some progress, tracked it down to Topaz which runs back shell code, I might be able to provide shell provider for it, till then I was able to reduce the code to:

bin/maglev-ruby -I. -Ilib/maglev/gems/1.8/gems/rake-0.9.2/lib -rrakelib/gemstone_env.rb -rrmaglev_stone.rb -rrake -rrake/file_utils -e "include FileUtils; Stone.new('maglev').topaz_commands([\"omit resultcheck\", \"run\", \"RubyContext createSmalltalkFFIWrappers\", \"%\"])"
@mpapis

I was able to find my way around all the rake tasks , but there is one that seams hard - https://github.com/MagLev/maglev/blob/master/rakelib/contrib/ottobehrens/stone.rb#L194-L200 - any ideas if ERB is needed? could I just find/replace it with sed?

@mpapis

@Monty ok the above commit is translation - no tests I'm even not sure if all is properly translated, especially the % does not seam as valid when I just throw at topaz, but maybe it's valid when it's put in context ...

the ruby/rake code was mostly hard to read because it was written for a lot bigger task and when used just for installation I could cut it down

@mpapis

@Monty ping - any progress on reviewing this code ... I know it's big and might seam messy, let e know if you need more details about it or some clarifications.

@timfel
Owner

I'll try and run this code on a few different systems here to give some feedback.

@timfel
Owner

Hi, while we're at it, can you add better error checking during the update process? So, downloading of the Gemstone VM failed for me the first time, but the script carried on trying to untar and install. It should stop where the error is instead.

@timfel
Owner

and i have a problem on one server, where the old update script downloads the gemstone package but this one doesn't. i will investigate

@timfel
Owner

the error is this: [Error] execution failed for '/usr/bin/wget http://glass-downloads.gemstone.com/maglev//home/tim/Devel/maglev/../GemStone-29699.Linux-x86_64.tar.gz' with error 8.

@timfel
Owner

and i think the fix is to replace gss_file with gss_name in the construction of the url.

@timfel
Owner

and now the url is missing the tar.gz extension - can you add that, too? :-)

@mpapis

sorry for that

@timfel
Owner

Now I'm getting this error: ./update.sh: line 244: build_maglev: command not found

@mpapis

I was testing this ... I guess refactoring before committing deserves another round of testing - sorry for the wasted time

@timfel
Owner

I had to make a couple more changes before this worked for me. Can you add those?

diff --git a/build_functions.sh b/build_functions.sh
index 96cb4d8..0278450 100755
--- a/build_functions.sh
+++ b/build_functions.sh
@@ -53,7 +53,7 @@ function cp_template()
     -e "s#<%= tranlog_directories.join(\",\") %>#$GEMSTONE_DATADIR/tranlog,$GEMSTONE_DATADIR/tranlog#" \
     -e "s#<%= FILEIN_DIR %>#$FILEIN_DIR#" \
     -e "s#<%= KEYFILE %>#$MAGLEV_HOME/etc/maglev.demo.key#" \
-    < erb_file > dest_file
+    < "$erb_file" > "$dest_file"
 }

 function maglev_stone_create()
@@ -65,7 +65,7 @@ function maglev_stone_create()
   mkdir -p "${GEMSTONE_DATADIR}/extent"
   mkdir -p "${MAGLEV_HOME}/log/${STONENAME}"
   mkdir -p "${GEMSTONE_DATADIR}/tranlog"
-  cp "${MAGLEV_HOME}/bin/extent0.ruby.dbf") "${GEMSTONE_DATADIR}/extent0.ruby.dbf"
+  cp "${MAGLEV_HOME}/bin/extent0.ruby.dbf" "${GEMSTONE_DATADIR}/extent0.ruby.dbf"
   chmod 0660 "${GEMSTONE_DATADIR}/extent0.ruby.dbf"
 }

@mpapis

and it worked? I'm so happy :)

@timfel
Owner

there's something else wrong, still (https://travis-ci.org/timfel/maglev/builds/4121309/#L63). I'm currently trying to get the travis build green on my repository, once I've got that, I'll enable it for this repo and we can have travis check this PR

@mpapis

now it should be more clear where it fails

@timfel
Owner

If you push again to nudge travis, it should test this PR.

@mpapis

added option --stonename so multiple gemstones can be supported ... not sure if it will be respected everywhere

@mpapis

@timfel ok that shows now there was error but I see no indication of errors in the log, is there a reason why topaz would return non 0 status and display no information about the problem? => https://travis-ci.org/MagLev/maglev/jobs/4128127/#L78

@timfel
Owner

Not sure what's going on here, maybe @Monty has an idea

@Monty
Collaborator

@mpapis - sorry I've been away. For some reason GitHub hasn't been mailing me when this issue is updated. I'll try and catch up on some questions from your previous posts.

@Monty ok the above commit is translation - no tests I'm even not sure if all is properly translated, especially the % does not seam as valid when I just throw at topaz, but maybe it's valid when it's put in context ...

run
...
%

The run command in topaz begins a block of GemStone/Smalltalk code. % ends it, which should get you back to a topaz prompt.

@Monty
Collaborator

@mpapis I don't know why there was no output from topaz. I'd need to see the complete topaz code that was input. It seems any redirection such as "output push" at the top level would be visible. It may well be the filein inputs other files that produce different log files than runstone_build_maglev_fileinruby.log.

[INFO] maglev build filein
78[ERROR] failed running 'build_maglev_fileinruby', last 10 lines of '/home/travis/builds/MagLev/maglev/fileintmp/runstone_build_maglev_fileinruby.log':

@Monty
Collaborator

github was mailing my old vmware address. I fixed that so I expect I'll get email updates going forward.

@mpapis

@Monty this are the logs I got on my local machine https://gist.github.com/9f94a593bde9848628d2 - the problem seams the same for me as in travis

@mpapis

ping? could someone review and let me know what to do with it?

@mpapis

@johnnyt ping me when you have time, I will be occupied with some extra activities for next 2 weeks, but I should be able to find time for you anyway skype:michal_papis

@mpapis

another half year later ... anyone?

@johnnyt johnnyt referenced this pull request
Merged

Improved installer #370

@jc00ke jc00ke closed this in #370
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Commits on Dec 12, 2012
  1. @mpapis
  2. @mpapis
Commits on Dec 13, 2012
  1. @mpapis
  2. @mpapis
  3. @mpapis
  4. @mpapis

    allow running stwrappers, restore rake, move copy to the optimal loca…

    mpapis authored
    …tion, do not use rake for start
  5. @mpapis

    cleaning and formating setup.sh

    mpapis authored
Commits on Dec 14, 2012
  1. @mpapis

    translate rake tasks to shell

    mpapis authored
Commits on Jan 11, 2013
  1. @mpapis
  2. @mpapis
  3. @mpapis
  4. @mpapis
Commits on Jan 13, 2013
  1. @mpapis

    changes from @timfel

    mpapis authored
  2. @mpapis
  3. @mpapis
  4. @mpapis
  5. @mpapis

    just reordering functions

    mpapis authored
  6. @mpapis
  7. @mpapis
Commits on Jan 14, 2013
  1. @mpapis
Commits on Jan 16, 2013
  1. @mpapis
  2. @mpapis
Something went wrong with that request. Please try again.