java-buildpack incompatible with the use of cf scale -MEMORY cmd

[gberche-orange]

The java-buildpack statically applies memory heuristics at staging time, i.e. includes the -Xmx option in the start cmd. As a result, if a user is using the cf scale -memory command to decrease the memory, the JVM will still try to allocate up to the initial memory account. The jvm will then exceed the memory limit and be killed by the warden container.

Possible workaround: apply the memory heuristics within the droplet using the http://docs.cloudfoundry.com/docs/using/deploying-apps/environment-variable.html#VCAP_APPLICATION limits entry.

[glyn]

For the new Java buildpack, we considered calculating the memory heuristics at droplet runtime (before we had access to $MEMORY_LIMIT at staging time), but this is pretty nasty because of the lack of reasonable maths precision in shell scripts. It would be necessary to add Ruby to the droplet, which seems like bloat.

A better solution would be to somehow get the buildpack to recalculate the memory limits after cf scale, but I understand that's a larger design change.

We could consider if there are ways to precalculate a guard which could run at droplet runtime to avoid excessive memory consumption, although I have no idea whether that's feasible.

On 5 Sep 2013, at 17:22, Guillaume Berche notifications@github.com wrote:

The java-buildpack statically applies memory heuristics at staging time, i.e. includes the -Xmx option in the start cmd. As a result, if a user is using the cf scale -memory command to decrease the memory, the JVM will still try to allocate up to the initial memory account. The jvm will then exceed the memory limit and be killed by the warden container.

Possible workaround: apply the memory heuristics within the droplet using the http://docs.cloudfoundry.com/docs/using/deploying-apps/environment-variable.html#VCAP_APPLICATION limits entry.

—
Reply to this email directly or view it on GitHub.

[gberche-orange]

I understand from http://docs.cloudfoundry.com/docs/running/architecture/stacks.html that ruby 1.9.3http://docs.cloudfoundry.com/docs/running/architecture/stacks.html%20that%20ruby%201.9.3 is indeed available at runtime in the default stack. I guess then that this would be a matter of running a small ruby program (a module/part of the buildpack) in the start command prior to excuting the jvm that would re apply the memory heuristics. Does this seem feasible ?

---

Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.

This message and its attachments may contain confidential or privileged information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
Thank you.

[glyn]

Yes, that seems feasible. We'd still want to run the heuristics at staging time because they emit useful warnings in addition to setting the memory sizes. But we could re-run the latter part of the calculation at droplet execution time. Alternatively, we could generate a shell script mapping feasible $MEMORY_LIMIT values to memory settings if we want to avoid the Ruby dependency.

I'll raise a feature for gracefully supporting scaling down. Thanks for raising this.

On 6 Sep 2013, at 08:07, Guillaume Berche notifications@github.com wrote:

I understand from http://docs.cloudfoundry.com/docs/running/architecture/stacks.html that ruby 1.9.3http://docs.cloudfoundry.com/docs/running/architecture/stacks.html%20that%20ruby%201.9.3 is indeed available at runtime in the default stack. I guess then that this would be a matter of running a small ruby program (a module/part of the buildpack) in the start command prior to excuting the jvm that would re apply the memory heuristics. Does this seem feasible ?

---

Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.

This message and its attachments may contain confidential or privileged information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
Thank you.

—
Reply to this email directly or view it on GitHub.

[nebhale]

@gberche-orange Just to let you know that we haven't forgotten about this, it's on our backlog but the push to PivotalCF release has de-prioritized it since you raised it. We're still committed to looking at it, but the introduction of memory bounding raises some issues around a reasonable implementation. When we've had a chance to fully dig into the issue, we'll let you know what we find.

[glyn]

I have prototyped running the memory calculation during droplet start and it works in the "golden path" cases (scaling memory up or down when there is enough memory) but produces extremely poor diagnostics in one error case (scaling down so that there is insufficient memory to cover the configured lower bounds) and has some new, worrying failure modes and fragility. We are currently considering our options.

Here's my latest brain dump on the subject in case @gberche-orange or anyone else would care to comment...

The only clean solution I am aware of is to have Cloud Foundry re-drive staging when memory is scaled up or down. This behaviour may need to be made conditional, e.g. by adding a setting to the hash returned by the release script, so that the Java buildpack could request to be re-driven when scaling occurs. It is not clear that the Cloud Foundry team would agree to provide such function as it would (currently) only be useful to the Java buildpack.

The current function (on master) is really unacceptable as scaling memory down too far will provoke warden to kill the application which currently produces very poor diagnostics. Although we hope the warden diagnostics will be improved, it is questionable whether they will ever be sufficient for this use case. Apart from poor diagnostics, a major downside of the current behaviour is that the application will not necessarily fail when it is scaled down too far, but may be killed by warden at an essentially arbitrary point in the life of the application.

The prototype on branch 56495606-graceful-memory-scaledown supports scaling memory up and down and causes the application to fail to start when memory is scaled down too far. If insufficient memory is available when the application is pushed, staging fails with suitable diagnostics much as on master.

The downsides of this implementation include:

• staging_info.yml no longer includes the JVM memory settings (although we could write these to a file and include the path of that file in stagine_info.yml). Users are already having to trust the buildpack to get the JVM memory settings right and this implementation removes the settings one step further away from the user's gaze.
• the start command returned by the release script now includes a backtick invocation of memcalc which means that:
  • if this invocation fails for any unexpected reason, then the JVM will run with default memory settings
  • if memcalc detects that $MEMORY_LIMIT is too small, it has to resort to returning "-Xmx0k" to cause the application to crash - this is retried by CF but the app ultimately crashes, which is acceptable, but with no accessible diagnostics, which is unacceptable.
• a significant proportion of the buildpack Ruby code is copied into the droplet which (a) pre-req's a suitable Ruby runtime in the DEA (which is otherwise superfluous to a Java application), (b) adds to the set of environments in which the buildpack code must run, and (c) complicates staging if we want to avoid adding all the buildpack Ruby code to the droplet.

This prototype might become a little more attractive if Cloud Foundry gathered logs during application start and delivered them back to the user of (g)cf push.

[gberche-orange]

Thanks @glyn for this prototype!

I agree with you that it would be best to have the buildpack contract refined, such as the release command provide additional information on whether the release command supports scale up/down dynamically or requires a restaging. Let's open the conversation on the vcap-dev mailing list.

Other buildpacks might suffer from the same issue (php, python...) Some runtimes, in particular those having GC have support for limiting the max memory in order to provide clear diagnostics when memory is low.

[nebhale]

We've come to the end of the prototyping effort here and determined that improvements need to be made to Cloud Foundry core. For posterity the prototype can be found at 8353fe2.

[kelapure]

@nebhale What are these improvements to core CF and when are they expected to be made ?

[nebhale]

@kelapure The improvements would be around better diagnostics when failures happen during startup of the a droplet, better diagnostics when applications are killed by Warden, and an enhanced buildpack API allowing either a second tier-of staging execution (that can occur without a "re-stage") or the ability for a buildpack to opt-into re-stages for certain lifecycle events.

To my knowledge there is no plan to make any of the changes described herein. Discussion about what changes might be made (not when) is ongoing in the vcap-dev list.

[kelapure]

Understood. Will the upcoming Diego project help ?

On Wed, Jan 22, 2014 at 6:12 AM, Ben Hale notifications@github.com wrote:

@kelapure https://github.com/kelapure The improvements would be around
better diagnostics when failures happen during startup of the a droplet,
better diagnostics when applications are killed by Warden, and an enhanced
buildpack API allowing either a second tier-of staging execution (that can
occur without a "re-stage") or the ability for a buildpack to opt-into
re-stages for certain lifecycle events.

To my knowledge there is no plan to make any of the changes described
herein. Discussion about what changes might be made (not when) is
ongoing in the vcap-dev list.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/8#issuecomment-33012633
.

[nebhale]

Some of our pain points are addressed in the Diego presentation, although sometimes making them worse. My understanding is that the staging changes in Diego are quite soft at the moment and need lots of input. I'm local to the Diego team next week and will talk about what improvements can be made with them then.

[kelapure]

Thank you any guidance to buildpack developers and implications for
buildpack particularly around staging will be appreciated!

On Wed, Jan 22, 2014 at 10:16 AM, Ben Hale notifications@github.com wrote:

Some of our pain points are addressed in the Diego presentation, although
sometimes making them worse. My understanding is that the staging changes
in Diego are quite soft at the moment and need lots of input. I'm local to
the Diego team next week and will talk about what improvements can be made
with them then.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/8#issuecomment-33031982
.

[gberche-orange]

@nebhale do you think the work on restaging has addressed the issue, i.e. should the new restage endpoint be called offered to be called upon a scale up/down by the cli ?

https://www.pivotaltracker.com/s/projects/966314/stories/65335102
https://groups.google.com/a/cloudfoundry.org/d/msg/vcap-dev/Oq-7XXwT7fs/0rq096cUWacJ

[nebhale]

To be honest, I don't have a lot of hands on experience with the new restage behavior so I can't say for sure. Restaging should be sufficient to handle changes to the memory allocated to an application, so I'd tentatively say yes, this issue should be addressed via restage.