The stomp gem was only used to support MCollective #66

Merged
merged 2 commits into from Feb 22, 2013

Conversation

Projects
None yet
2 participants
Contributor

slippycheeze commented Feb 20, 2013

We only included the stomp gem in the system to support MCollective, which is
now gone from the Microkernel. This removes the gem as well, for a marginal
reduction in size, and less things to care about when figuring out what works
or not in the MK image.

This closes #65.

Signed-off-by: Daniel Pittman daniel@rimspace.net

The stomp gem was only used to support MCollective
We only included the stomp gem in the system to support MCollective, which is
now gone from the Microkernel.  This removes the gem as well, for a marginal
reduction in size, and less things to care about when figuring out what works
or not in the MK image.

This closes #65.

Signed-off-by: Daniel Pittman <daniel@rimspace.net>

tjmcs commented Feb 20, 2013

This is close, but it might not be a complete fix for the issue (depending on how you view things). There is still an edge case that this doesn't quite resolve. If you are using the -r or --reuse-prev-dl flag (short or long form of the same flag) when building the bundle file and the previous build was done prior to this change, then the gem is not installed during the boot process, but it is bundled into the ISO (so it's there, taking up space, even though it's not installed during the boot process). I'm not sure if that's important enough to hold up this merge or if we just want to accept that, at least for this edge case, there might still be a bit of extra space taken up by the gemfile in the ISO (even though the gem isn't actually installed)...

Thoughts?

Remove the dangerous "reuse previous download" option
The script that collected data together for the Microkernel image had a "reuse
previous download" option that triggered a particularly dangerous set of
behaviours:

When enabled it assumed that it should simply use the existing build directory
as-is, and should avoid fetching content that already existed in
that directory.

This seems innocuous enough - an optimization you might choose to use during
development - but actually masks a dangerous failure mode.  If we eliminate
something from the build, like we did the stomp gem, it may still be built
into the final image when reusing the downloaded content, because it happens
to sit in the build directory.

A correct version of this would use a cache that sits aside from the build
directory, and only put in place files that would actually be used.

That is more complex to develop, and doesn't really add value compared to
using a simple squid proxy server to cache the downloaded files, so I have
simply expunged the reuse facility.

Signed-off-by: Daniel Pittman <daniel@rimspace.net>
Contributor

slippycheeze commented Feb 20, 2013

Tom, you are absolutely correct. That wouldn't work correctly. Thanks for identifying it.

I have added a commit to expunge this dangerous feature, so that nobody is tempted to use it - it requires expert knowledge of what changes have happened, and how to make them safe, which isn't a good bet for anyone. Human error is just too common to trust this feature as-is.

If you think it is important enough I would accept a follow-up to add back a correct local caching option that didn't suffer this flaw, but using a simple squid cache definitely solves any performance issues we have local.

tjmcs commented Feb 21, 2013

Actually, I use this feature all the time (my internet connection here at home is none to speedy, and so it can take 10 minutes or more to build an ISO if I'm not reusing the downloads, versus less than a minute if I am).

I'd prefer to leave the flag in place and just make sure that the gem file in question is removed if it's present...I understand why you might not want to use that flag for your CI-based builds, but for testing locally it's a real time saver...especially when I have to iterate through the build process over and over (like I've had to do at times when debugging issues with an ISO).

Contributor

slippycheeze commented Feb 21, 2013

I highly recommend that you either invest the ten minutes it would take to set up a squid cache - and get the same minute builds without the flag - or invest in fixing the problems with the model that it implies.

I felt the same pain when I was getting the CI system working - ibiblio is very much outbound limited, so I only saw ~ 40kb/s downloads. Using the squid proxy turns that into instant response on every build, even without the reuse flag.

I don't think that working around problems with your networking by allowing users to pass a "please break things" flag is a good model to retain. I don't mind a version that works correctly, but I am very, very reluctant to keep this just to reduce setup cost at your end.

tjmcs commented Feb 21, 2013

Let me think about that for a bit before we merge this change...it's a flag that I've used a lot, but I've also always worried about differences in builds if my "reused components" are different from yours for some reason...the flip side of that is that the versions that are included in any given build could change depending on when a given ISO was built (if an external component was updated remotely between builds)...

I'll play around with setting up a squid cache in the VM that I use for testing these builds, but let me give the removal of this flag from the bundle build file some thought...

tjmcs commented Feb 21, 2013

@daniel-pittman, do you have notes on how you configured your squid proxy (to proxy for just the sites used in building the bundle file?)...I'm on a somewhat resource-constrained VM, so it would be nice to not have to proxy all HTTP requests (and just proxy the ones used in building the bundle file)...

If you've got notes on what you did, I'd really like to get that into the Wiki somewhere before we pull this flag out of the build script (so that other Microkernel developers can also set up their own proxy to aid in the reduction of bundle build times)...thoughts?

Contributor

slippycheeze commented Feb 21, 2013

@daniel-pittman, do you have notes on how you configured your squid proxy
(to proxy for just the sites used in building the bundle file?)...I'm on a
somewhat resource-constrained VM, so it would be nice to not have to proxy
all HTTP requests (and just proxy the ones used in building the bundle
file)...

  1. Install squid
  2. `export http_proxy=http://proxy-machine.local:31280/
  3. ./build-bundle-file.sh ...

The "out of the box" configuration of squid worked just fine for me. :)

If you've got notes on what you did, I'd really like to get that into the
Wiki somewhere before we pull this flag out of the build script (so that
other Microkernel developers can also set up their own proxy to aid in the
reduction of bundle build times)...thoughts?

I will update the wiki now, but again, this feature is broken right
now. It will not work correctly unless those developers are expert
enough to know that they have to manually remove the stomp gem, or to
make a "non-reuse" run, because of this change.

With that change are you happy to see this merged?

tjmcs commented Feb 21, 2013

I understand your concerns (and have had the same ones myself). That flag was added as a convenience, to make the build process faster. If we can document how to use squid to accomplish the same thing, then I think I'm good to go...

So you haven't to modified the default squid config at all? I'd been fighting with trying to set things up so that squid would handle only the 'wget' requests bound for the domains for the servers involved in the build-bundle-file.sh script (with no luck). I really don't want to proxy all HTTP requests on this machine, just the ones involved in the bundle build process...if I understand the default configurations correctly then squid will be the proxy for everything (including the HTTP requests made to the Razor server on this same machine?)...

Contributor

slippycheeze commented Feb 21, 2013

I understand your concerns (and have had the same ones myself). That flag
was added as a convenience, to make the build process faster. If we can
document how to use squid to accomplish the same thing, then I think I'm
good to go...

nod If squid wasn't pretty much a drop-in replacement, I would have
thought more about fixing the feature than dropping it.

So you haven't to modified the default squid config at all? I'd been
fighting with trying to set things up so that squid would handle only the
'wget' requests bound for the domains for the servers involved in the
build-bundle-file.sh script (with no luck). I really don't want to proxy all
HTTP requests on this machine, just the ones involved in the bundle build
process.

nod We only set the proxy environment variable in the shell that
runs the build process, so it is only used when downloading
Microkernel content. If you leave http_proxy unset, etc, then none
of your tools will talk to Squid.

You could also just set / export the proxy in the
build-bundle-file.sh script, but env http_proxy=... ./build-bundle-file.sh seems simpler overall. :)

..if I understand the default configurations correctly then squid
will be the proxy for everything (including the HTTP requests made to the
Razor server on this same machine?)...

Unless you set the environment variable (and, in Ruby, ask explicitly
for it to be used), nothing will use the squid proxy, so you shouldn't
have any trouble there.

tjmcs commented Feb 21, 2013

Could we replace the flag we're removing with a flag that could be used to set the "http_proxy" variable? Would then make things pretty much self-contained…

I've also noticed that there are a number of dependency files that the build-bundle-file.sh script tries to load that don't exist…the result when using the squid proxy is that these are the slow parts of the bundle build process (since you have to go out to the server to try to find them even though they don't exist)…is there any way to resolve that in squid?

On Feb 21, 2013, at 12:11 PM, Daniel Pittman notifications@github.com wrote:

I understand your concerns (and have had the same ones myself). That flag
was added as a convenience, to make the build process faster. If we can
document how to use squid to accomplish the same thing, then I think I'm
good to go...

nod If squid wasn't pretty much a drop-in replacement, I would have
thought more about fixing the feature than dropping it.

So you haven't to modified the default squid config at all? I'd been
fighting with trying to set things up so that squid would handle only the
'wget' requests bound for the domains for the servers involved in the
build-bundle-file.sh script (with no luck). I really don't want to proxy all
HTTP requests on this machine, just the ones involved in the bundle build
process.

nod We only set the proxy environment variable in the shell that
runs the build process, so it is only used when downloading
Microkernel content. If you leave http_proxy unset, etc, then none
of your tools will talk to Squid.

You could also just set / export the proxy in the
build-bundle-file.sh script, but env http_proxy=... ./build-bundle-file.sh seems simpler overall. :)

..if I understand the default configurations correctly then squid
will be the proxy for everything (including the HTTP requests made to the
Razor server on this same machine?)...

Unless you set the environment variable (and, in Ruby, ask explicitly
for it to be used), nothing will use the squid proxy, so you shouldn't
have any trouble there.

Reply to this email directly or view it on GitHub.

Contributor

slippycheeze commented Feb 21, 2013

Could we replace the flag we're removing with a flag that could be used to
set the "http_proxy" variable? Would then make things pretty much
self-contained…

We could, but it means writing a bunch of code to duplicate an
existing feature of the Unix shell. I guess I don't mind if you want
to do that, but it seems like wasted effort to me, since it is
actually less typing to configure it using the shell too. :)

I've also noticed that there are a number of dependency files that the
build-bundle-file.sh script tries to load that don't exist…the result when
using the squid proxy is that these are the slow parts of the bundle build
process (since you have to go out to the server to try to find them even
though they don't exist)…is there any way to resolve that in squid?

http://www.squid-cache.org/Doc/config/negative_ttl/ should do the right thing.

tjmcs commented Feb 21, 2013

Yeah, saw that parameter in the config notes but I haven't had time to play with it. The default value is "0 seconds"...I assume you could change this to something longer, but I was starting to wonder if we should be smarter about downloading the dependency files themselves (perhaps downloading dependency files for the TCEs that have a dependency file instead of blindly trying to download a dependency file for each extension on our list?)...

As far as a proxy-server flag goes, I'm happy to add that to the script if you don't want to modify your current pull request to include it. I don't think it's that much work, and I do think it gives users the ability to set a proxy for the HTTP requests if they want to do so. Yes, I could export a variable (as long as I remember to unset it later, or close the shell), and yes, I could do it from the command line as part of the same command (if I'm using a shell that supports the syntax). I could even write a wrapper script that would set the proxy for me before invoking the build-bundle-file.sh script. I just think it'd be easier for some users to set it as a CLI argument to the script...

Contributor

slippycheeze commented Feb 21, 2013

Yeah, saw that parameter in the config notes but I haven't had time to
play with it. The default value is "0 seconds"...I assume you could change
this to something longer, but I was starting to wonder if we should be
smarter about downloading the dependency files themselves (perhaps
downloading dependency files for the TCEs that have a dependency file
instead of blindly trying to download a dependency file for each extension
on our list?)...

It could be. I would certainly accept a change that improved that, if
you wrote it. Can't see a downside risk.

As far as a proxy-server flag goes, I'm happy to add that to the script if
you don't want to modify your current pull request to include it.

Sure. I don't object to it being added, if you think it adds enough
value to spend the time writing it.

tjmcs commented Feb 21, 2013

Looks like the lag I'm seeing has nothing to do with missing flags...for some reason my squid proxy server hangs on some requests (always the same ones). I'm going to start over from scratch (with a clean cache) in case something got corrupted. The only thing I changed in the default config was the location of the (ufs) cache directory and it's size...will let you know how it goes

Contributor

slippycheeze commented Feb 22, 2013

OK. In that case, are we happy to merge?

tjmcs commented Feb 22, 2013

Left you a voicemail; still having issues with trying to run squid on my "test VM" to make up for the flag that you're removing in this pull request...

I haven't been able to track down the issue. Have doubled the default cache_mem, doubled the memory available to the VM, and even rebuilt the cache completely (in case something was corrupted there somehow). In every case, on the 35th request, the build-bundle-file.sh script hangs for about 5 minutes waiting for that request, then continues, then hangs again, then continues...

It always hangs on the same line and always takes about the same amount of time. Basically the squid proxy is reducing my 10 minute build times (without the proxy) to 9 minute build times. Not exactly the improvement I was hoping for...

Unless we can come up with a reasonable (and documented) proxy solution to make up for the flag that's being pulled out in this pull request, I'm not happy with removing the flag. Waiting 27 minutes to build three ISOs instead of 30 minutes really isn't reasonable...

Can we chat about the configuration you're using in your CI environment (version of squid, platform, etc.) to see why you're not having any issues and it's giving me nothing but grief?

slippycheeze added a commit that referenced this pull request Feb 22, 2013

Merge pull request #66 from daniel-pittman/feature/master/65-remove-u…
…nused-stomp-gem

The stomp gem was only used to support MCollective

@slippycheeze slippycheeze merged commit 26a1a44 into puppetlabs:master Feb 22, 2013

Contributor

slippycheeze commented Feb 22, 2013

I haven't been able to track down the issue. Have doubled the default
cache_mem, doubled the memory available to the VM, and even rebuilt the
cache completely (in case something was corrupted there somehow). In every
case, on the 35th request, the build-bundle-file.sh script hangs for about 5
minutes waiting for that request, then continues, then hangs again, then
continues...

Unless we can come up with a reasonable (and documented) proxy solution to
make up for the flag that's being pulled out in this pull request, I'm not
happy with removing the flag. Waiting 27 minutes to build three ISOs instead
of 30 minutes really isn't reasonable...

The reasonable solution is to use any HTTP proxy in ... honestly, just
that. Nothing special. I have no idea why you are unable to get
vanilla squid to operate out of the box, but whatever the cause it is
something local to your system and/or configuration and/or
distribution.

Can we chat about the configuration you're using in your CI environment
(version of squid, platform, etc.) to see why you're not having any issues
and it's giving me nothing but grief?

Setting this up is trivial:

  1. Install squid.
  2. Start squid running.
  3. Set http_proxy to point to it so that wget uses it.

...and I am not kidding. This isn't "specially tuned and modified
squid", this is "install squid and it works".

For help debugging squid you should probably contact your
distribution, or the upstream squid folks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment