Runtime hooks

[thomaseizinger]

I am having problems understanding the runtime hooks documentation at https://docs.run.pivotal.io/devguide/deploy-apps/deploy-app.html#profile.

As far as I understand the documentation, I have to put the .profile file into the root of my application directory.

My directory structure looks like this when I SSH into the application:

├── app
│   ├── .java-buildpack // omitted
│   └── rest-api-0.0.3
│   ├── bin
│   │   ├── rest-api
│   │   ├── rest-api.bat
│   │   ├── run-minimal.bat
│   │   └── run-minimal.sh
│   ├── docs
│   │   └── index.html
│   ├── lib // omitted
│   └── .profile <-- this is my .profile file
├── .bash_logout
├── .bashrc
├── logs
├── .profile
├── staging_info.yml
├── tmp
└── tree

But the file does not get executed. My .profile file looks like this:

echo "This is a test." > mytestfile

If it got executed, I would expect a file mytestfile anywhere in this directory structure.

The .profile folder outside of my app directory is not under my control, the only thing the cf CLI pushed to the server were the contents of the app/ folder.

Where do I have to place the .profile file in order to get executed?

I am using the java-buildpack and my API endpoint is https://api.run.pivotal.io (API version: 2.61.0).

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/130517887

The labels on this github issue will be updated when the story is started.

[sclevine]

Hi @thomaseizinger,

Unless @nebhale is aware of any recent changes, I do not think it is possible to use a .profile file with the Java buildpack. The .profile file needs to be in the root of the folder you push the application from (and will end up at the same level as staging_info.yml in the container in the app folder). There is no way to do this with the Java buildpack, which requires you to push a JAR or WAR file. You could use cf set-env or an application manifest.yml file instead.

The documentation is confusing, because it sets Java-related environment variables in the example. @jncd, can you update that page to use non-Java environment variables and add a note mentioning that it is not possible to use a .profile file with the Java buildpack?

-Stephen

[nebhale]

The Java Buildpack makes no special considerations for a .profile file in the root of an application. I'm not sure if the support in other buildpacks is something from the container or the buildpack itself; if it's the container, then you should be able to get it without any explicit support from the buildpack, if not, it's unlikely that we'll add support for it.

Whether or not there is support for it, I highly discourage using it. Including environmental configuration as part of an application is clear violation of the 12 Factor principals that separate application from the environment that it runs in. By tying the environment to the application (e.g. including a .profile), you're making the application less portable and more tied to deployment environment. The fact that you're having trouble now illustrates this exact problem. Your applications should expect to start in any environment, whether in your IDE, locally on a development machine, on a bare-metal production server, in a virtual machine, or even in a Cloud Foundry container without needing any special support. Applications should depend only on their environment being properly configured, they should not be configuring that environment on their own.

[animatedmax]

I've created a story in the CF Docs (Public) Tracker to update the Java Buildpack topic:
https://www.pivotaltracker.com/story/show/131001619

[jncd]

Fixed error on page: https://docs.run.pivotal.io/devguide/deploy-apps/deploy-app.html#profile
(Might need to wait a day or two to see it published.) Thanks @thomaseizinger for alerting us. And, thanks to others for answering.

[thomaseizinger]

Thank you very much for your answers.
@nebhale The 12-factor app config principle was actually the reason why I want to do this. I totally understand your point, that is why I actually need the .profile file. I want my app to be unaware that it will ever run in the cloud foundry environment. However, the structure of the VCAP_SERVICES environment variable couples my app tightly to the cloudfoundry environment.
The point is, if I want my application to always connect to the database given through the environment variable, I have two choices:

1. Create a JSON-encoded environment variable that mimics the structure of VCAP_SERVICES in every environment I have:
   This is pretty cumbersome because in the end, I would probably have to rely on some config file in every environment but that would totally miss the point of the 12-factor app config principle.
2. I have to make my app unaware of the VCAP_SERVICES variable:
   This was the route I planned on going and that is why I need the .profile file because it would actually just contain this line:
   export DATABASE_CONNECTION_STRING=$(echo $VCAP_SERVICES | ./lib/jq .elephantsql[0].credentials.uri' | cut -d "\"" -f 2)

jq is a handy small tool that lets you do pretty awesome stuff with JSON but in this case, I am only using it to extract the connection string to another environment variable which I then use in my application to connect to the database. This way, my app is unaware of the cloudfoundry environment and it is way easier to launch the application locally, because I simply have to set the DATABASE_CONNECTION_STRING environment variable and that is it.

I would rather have a .profile file hidden somewhere in the application instead of coupling my app to a JSON-structured environment variable. In addition, not having to parse the JSON in my application also makes the code simpler and less error prone.

Are there any other hooks in cloudfoundry where I could define this logic?

[sclevine]

I agree that there are valid use cases for .profile support that don't violate the 12-factor config principal. For instance, cf set-env and manifest.yml don't allow code to be evaluated before setting environment variables, and not all environment variables are used for configuration.

It's also worth pointing out that .profile support is a built-in CF concept that was introduced fairly recently (CF v238). Here's the original story to add support. The motivation for .profile support does appear to be similar to the use case that @thomaseizinger describes: supporting applications that don't account for CF.

@nebhale how opposed would you be to allowing CF to evaluate a .profile file that was included at the root of JAR or WAR file?

[sclevine]

(Also, @thomaseizinger, FYI: jq is included in the rootfs.)

[thomaseizinger]

@sclevine, thank you for putting together all this information. I already looked into some of these links and I am also very interested in finding a solution for this problem.
Addressing jq: Good to know jq is included, so I don't have to bundle it with my application, altough I am not sure yet, if bundling wouldn't make the application more portable because I don't have to worry about jq being installed. But that is a different story.
I am also not sure if this issue is still the right place to discuss this. The original issue of clarifying the documentation was resolved, so if it would be more appropriate to open an issue in another repository, just tell me.

[sclevine]

The original issue of clarifying the documentation was resolved, so if it would be more appropriate to open an issue in another repository, just tell me.

I'd like to keep this issue open until Ben chimes in. If he decides that the Java buildpack should support .profile, it might make sense for the docs team to update the notice to mention that support is forthcoming or to create a story about removing the notice for when the feature arrives.

[thomaseizinger]

Any news / progress on this?
Should I open a separate issue in the java_buildpack repository for a discussion on whether and how this requirement could be implemented?

[nebhale]

We have no plans to add any specific support for .profile to the Java Buildpack. Referencing the specific case of the DATABASE_CONNECTION_STRING, you shouldn't actually need a .profile
to achieve your goals there. That value can be set with cf set-env and evaluated as part of the standard shell around the Java process. We do this today to extract data from VCAP_APPLICATION and inject it into the configuration of various APM providers.

[thomaseizinger]

That helped a lot, thanks!
Just for the reference, I am now doing the following in my CI-configuration:

#!/usr/bin/env sh

[ -z "$1" ] && echo "App ID not given." && exit 1

cf set-env \
$APP_NAME \
DATABASE_CONNECTION_STRING \
$(cf curl /v2/apps/$1/env | jq '.system_env_json.VCAP_SERVICES.elephantsql[0].credentials.uri' | cut -d "\"" -f 2)

This is the content of a shell-script that is called before the app is pushed to CF via cf push.
As the content of the cf env command is not meant to be parsed by scripts, I directly call the underlying endpoint (/v2/apps/:uuid/env) of CF to fetch the configuration of my app. The result is piped to the jq utility and finally stripped from the surrounding quotes.

At least for me, this issue is now resolved.
Thanks everyone for the help!

[bentarnoff]

Thanks, @thomaseizinger. I'm closing this issue now that it's resolved.

[nebhale]

@thomaseizinger The other thing to remember is that you don't have to have the "final" result before the setting the environment variable. You can put in enough escaping to do something like:

cf set-env <APP_NAME> DATABASE_CONNECTION_STRING '$(echo $VCAP_SERVICES | jq \'.elephantsql[0].credentials.uri\')'

(I freehanded that, so it might not be quite right)

This means that the variable will be evaluated each time a container is created and if you rebind a service, you can restart the container without resetting the environment variable.

[thomaseizinger]

That works? Nice!
I tried setting the environment variable to a BASH-expression in the Web-UI but that did not work for me. Does this only work with the cf set-env command?

[nebhale]

I've never tried it from the web UI, so I'm not sure if it's supported there or not (but I'd suspect yes). They key is that it requires a lot of escaping as the value is evaluated three times, and only the last evaluation will resolve the proper value.

[thomaseizinger]

Are you sure that CF evaluates the value of the environment variable as a BASH expression before handing it over to the application?
I tried several ways and the closest I got was the following output of cf env <APPNAME>:

User-Provided:
DATABASE_CONNECTION_STRING: $(echo $VCAP_SERVICES | jq elephantsql[0].credentials.uri)

When trying to start the application, the log prints:

2016-10-12T20:39:51.580+02:00 [APP/PROC/WEB/0] [ERR] Exception in thread "main" java.lang.IllegalArgumentException: Illegal character in path at index 6: $(echo $VCAP_SERVICES | jq elephantsql[0].credentials.uri)

[nebhale]

I wouldn't expect see an evaluated version of the expression in cf env (as that is returned from the database, not from some container where it could be evaluated). That error though, might be indicative of a lack of quotes around the jq expression. Let me see if I can work out what you'd want to set.

[thomaseizinger]

I am not sure if quotes around the jq expression are the root of this problem.
I copied the above error from the log stream of my application, which simply displays the message of the IllegalArgumentException that was thrown but my application in the attempt of parsing the string given via the env variable DATABASE_CONNECTION_STRING as an URI.
To me, this would mean that CF passed the whole expression as string to my application.

As far as I know, most of the code that runs CF is written in Go? I am not sure how passing a BASH-expression should work as I don't see any BASH-scripts involved when running / creating an instance of an application. (Except the above mentioned .profile script.)

[mfeltscher]

Since I stumbled upon this issue while looking for a similar solution I wanted to share my solution with you as well. Because I don't know my app's GUID but only its name (represented by $APP_NAME in my example below) but still want to be able to access information from outside the app (in a CI build script for example) I came up with a workaround using cf ssh to obtain information about my bound services:

cf ssh $APP_NAME -c "echo \$VCAP_SERVICES" | jq -r ".elephantsql[0].credentials.uri"

This requires you to enable SSH in your CLI. Also note the -r option which tells jq to output the raw value so you don't have to use cut anymore.

Maybe this is helpful to someone else as well :)

[thomaseizinger]

Thanks for sharing! That is indeed a helpful (and neat) solution and way better than the way I currently handle the problem.
I wrote a (pretty ugly) plugin that actually does all this work. I called it "get-env" and given an app name and a JSON path, it outputs the found value. I planned on cleaning up the code and releasing it (and also share my solution here) but unfortunately, I did not yet have the time to do so.

It might also be a good idea to include this in the docs somehow?

[darren-steven]

We have a legacy application (binary) that we can launch using the java buildpack, but prior to tomcat running, we need to create a bunch of xml configuration (from VCAP_SERVICES) in the location where the app looks for it. being able to have a pre-run step would be very useful. I think multi buildpacks might solve it (or a fork of the buildpack). we can't use cf ssh in our instance as it's been explicitly blocked.

[drnic]

Java & Binary apps can now use .profile via https://github.com/starkandwayne/dot-profile-buildpack

[berndgoetz]

As of now it's included in the Java buildpack, it seems. Just include the .profile file in the jar or war file.