Can newExtent accept zipped extents?

[dalehenrich]

See this thread for discussion.

1. allow a zipped extent file as argument to newExtent
2. allow a backup file (zipped or not) to be used as argument to newExtent

[dalehenrich]

@marianopeck ... there is a complication when using a backup file --- which is actually a restore from backup issue more than a newExtent issue...

Basically I would like to start a restore from backup with an extent0.dbf --- in my particular use case for newExtent, the original extent may not be usable,s o I need to ensure that I start with an extent that can be used:

I use newExtent in new development all of the time, when doing a new build and the old extent is not always usable from the new version of stone executables ... of course in these cases, I'm also not very interested in using restore from backup, because the backup may not be usable either --- mainly in the area of possible primitive changes or changes to base methods that are "required'

it's also convenient to replace the extent with extent0.dbf so that you get a minimum sized repository after the restore (effective shrink db) ...

The complication comes because it is necessary to login into the initial extent as DataCurator/swordfish, but after the restore is complete you have to use the appropriate userid/password combination ..

I have an idea about how to solve this, but I thought I would ping you before doing too much fiddling around without getting more information about your particular use case and your thought about being able to restore from backup starting with a virgin extent0.dbf.

[dalehenrich]

With these last two commits (one for tODE and one for GsDevKit_home) I've implemented what appears to be a satisfactory solution for newExtent using a backup file whose userId/password (from sessionDescript for stone) is different from the extent being used as a starting point (in newExtent case that $GEMSTONE/bin/extent0.dbf using DataCurator/swordfish)

[marianopeck]

Hi @dale,

Yes, starting with a fresh extent0.dbf is indeed a good idea. I do this in a script I call shrinkStone which is basically a copydbf of the fresh extent0.dbx and then do a restoreFromBackup. That is how I shrink extent size.

And yes, for MY particular case, you should use DataCurator/swordfish to make the restore but after, use the user I specified in createStone via -P and -U.

Does that make sense? If true, this is cool because we can use createStone from a zipped backup file!

[dalehenrich]

@marianopeck if you pick up latest tODE and latest GsDevKit_home you should be able to test out the newExtent code ... -h help text should contain info

[dalehenrich]

I did like your suggestion of being able to do newExtent with a backup file ... so let's see if this will work for you ... BTW the GsDevKit_home code is on the issue_162 branch ...

[dalehenrich]

... need to add tests:)

[marianopeck]

Hi @dale.

OK, so I should merge issue_162 branch into gs_port right? and tODE in which branch?

Thanks!

[marianopeck]

@dalehenrich

I am not sure about your "if you pick up latest tODE". I can imagine installing latest tODE on my original stone, but I am not sure if that's what you mean. Maybe you mean just git-update the local filetree of tODE inside GsDevKit_home ?
Also..would I need to rebuild command line image?

Thanks in advance

[marianopeck]

Sorry and another question... I was using this script:

createStone -fg -U $stoneUsername -P $stonePass -s /quuve/sites/$sourceStoneName/gemstone/data/extent0.dbf $sourceStoneName 3.2.9

Soo... as far as I can see...you added the support into newExtent but not on createStone right? So...I must first call the newExtent and THEN createStone somehow (because now the extent is already created).

[dalehenrich]

@marianopeck sorry I had to head and take care of grandkids ... "picking up latest tODE" means that you do a git pull origin dev in the tODE repo (I assume that you were already on the dev branch?). To get the latest GsDevKit_home, you do the following:

cd $GS_HOME
git fetch --all
git checkout issue_162

Then you need to rebuild the devKitCommandLine image by doing the following: updateGsDevKit -d ... and that should be it ...

Finally, createStone, smalltalkCI, and upgradeStone all have -s options and the -s argument is passed through to the newExtent call in those other scripts --- at least it should:) --- I manually tested newExtent for the three combos: backup file (compressed or not), raw extent file, gzipped extent file.

When I create the tests I'll make sure that I have coverage for all of the scripts.

Tomorrow will be a busy day for me, so I probably won't be available much nor will I get much more done:)

[marianopeck]

Hi @dalehenrich

Thanks for this code. It's already useful. I was able to do a createStone to move my old stones into new gsDevKit_home based stones (3.2). However, I am not failing to do a upgradeStone to move those already gsDevKit_home 3.29 stones into 3.3.3 stones. I tried something like this:

       $GS_HOME/bin/upgradeStone -f -U "$stoneUsername" -P "$stonePass" -s "$GS_HOME/server/stones/$sourceStoneName/backups/quuve-forUpgrade.dbf.gz" "$sourceStoneName" "$upgradeStoneName" 3.3.3

That brings the error: snapshot file type ('expected') must be 'extent' if -n option specified ::

So, this way, I still need the extent in both places (in the original stone) and when doing upgradeStone it will try to do a copydbx with the extent.

Now, the error is thrown explicitly so I guess you have a reason for doing that. But on the other hand, the help comment of upgradeStone says backups are possible:

  -s <source-stone-extent-file-path>
     Path to  <source-stone-name> extent file used as starting point for
     upgrade. Path may be a relative file path. The snapshot file may be
     an extent file (raw or gzipped (.gz)) or a backup file.
     If the file is compressed, it will be uncompressed into the target
     extents directory. If the file is a backup file, a
     $GEMSTONE/bin/extent0.dbf will be copied into the target extents
     directory and then the backup will be restored without replaying
     tranlogs.

If backups are not allowed, then I could try with a compressed extent.

Thoughts?

[marianopeck]

Hi @dalehenrich

Of course, the reason of above is because upgradeStone does this:

else
  rm -rf $GS_SERVER_STONES/${targetStoneName}/extents/extent0.dbf
  $GS_HOME/bin/newExtent -n -s $snapshotFile $targetStoneName
fi

So..I am wondering...maybe the correct solution is to allow the -n to be specify in upgradeStone and passed to newExtent? In either case, does it make much sense to perform an upgrade with tranlogs?? I mean...if you are going to perform an upgrade I guess we can assume we have a correct extent/backup in which we would NOT need to recover from logs. Right?

[marianopeck]

OK. I tried removing the -n option from the newExtent call in upgradeStone and the problem I found is that when the copydbf is called with the sourced env file of 3.3.3 I get this error:

Copy snapshot and remove tranlogs
snapshot file: /home/quuve/GsDevKit_home/server/stones/debrisDemo/backups/quuve-forUpgrade.dbf.gz
copydbf: File is not compatible with this executable;
expected compatibility level = 846, found 844, in file /home/quuve/GsDevKit_home/server/stones/debrisDemo/backups/quuve-forUpgrade.dbf.gz
no pgsvr connection exists

ufffff.....so the newExtent should know which "product" to use as it might not be the same as the target one...uhhh..

Thoughts?

[dalehenrich]

@marianopeck ... haha, I was thinking about the upgradeStone situation after I sent email ...

I did change the -s help text to include backups as an option for upgradeStone, but after thinking about it I don't think it is possible to use the backup option for an upgrade and only extents can be used.

We really don't recommend or even support taking a backup from one version of GemStone and restoring it into another version of GemStone, so I think I need to change the help text for upgradeStone.

[marianopeck]

Hi @dalehenrich

Ok, got it. However, note that the error reported of copydbf (using it for a different gemstone version) could still happen even when using extents, right? Or when using that would not happen? Ok, I will discover myself but I must restart my work as I deleted the extent (and kept only the backup hahah).

Thanks!

[dalehenrich]

copydbf knows the difference between backups and extents. With that said, I've using copydbf for copying extents for upgrade for quite a while now and haven't seen a problem, although it does look like the installation instructions document using 'cp' ... if it turns into a problem, then we'll have to use cp:)

[marianopeck]

OK..and let me ask...in order for copydbf to know how to uncompress a compressed extent...which "format" / "tool" should I use? gzip I guess...

[dalehenrich]

Yes, I think the help text mentions gzip ... copydbf recognizes gzipped extents and backups ... I tried zip but copydbf won't copy zipped files ... it looks like lz4 compression may be understood by copydbf as well, but I'm not sure what bash command will read/write lz4 compressed files and I'd need to teach newExtent to decompress using the lz4 utility.

[marianopeck]

Hi @dalehenrich

OK, you are right, copydbf worked correctly with extents. And in general, this issue is fixed . I mean, at least, I can confirm the whole process did work for me. The only real problems are:

1. The one I reported in issue newExtent copydbf problem trying to get filetype #163
2. Remove the fact that backups are allowed in upgradeStone -s.