How to read static configuration from git repository without building data only container?

[Krieke]

From which Haufe-Lexware repo comes the content of http://wickedhaufeio.readthedocs.io/en/latest/

I thought it would be this repo, which I quess it is... but the readthedocs is not updated with the changes made 18 days ago. Anyhow: I am trying to run this service on DC/OS but building the wicked-api documentation is not sufficient with my experience. In the readthedocs there is a reference to docker-build.sh which I can't find in any of the Haufe-Lexware repos.

In other documentation I read the following: The STATIC CONFIG is created at deployment time by building a static configuration container directly on the docker host which is to host the API Portal. This is usually done by keeping a special Dockerfile inside the deployment repository, which in turn clones the configuration repository into this container.

So I indeed created a repository, but I have no clue which how to address this repo to the docker image on docker hub. Are do I have to write this intelligence my self... then I would be handy on how to build the wicked-api

Thx for your help and feedback.

Daniel

[DonMartin76]

Hey Daniel, yes, the source is this repository; I just realized I hadn't recreated the documentation on readthedocs in a while, have to look into automating that (I actually thought it would react to git changes, but apparently it doesn't).

Running all of these things "natively" (i.e. not in docker) is currently not actively supported, so there is usually no need to build your own wicked containers (they're published to hub.docker.com).

Check out this on how to use the Kickstarter to create your configuration repository:
https://github.com/Haufe-Lexware/wicked.haufe.io/blob/master/doc/creating-a-portal-configuration.md

And this on how to create a docker-compose.yml file which usually works out of the box for local deployments:
https://github.com/Haufe-Lexware/wicked.haufe.io/blob/master/doc/deploying-locally.md

When you have that up and running, it's usually a small step to get it up and running on an external docker host (it's just a question of which docker-machine is active, and the IP/DNS and certificate configuration).

If you get stuck anyplace, feel free to file an issue here (like this one). And in case of doubt, this repository is the source of truth.

Best regards,
Martin

[Krieke]

Hello Martin,

Thx already for your reply: That is just the case... I want to run the
wicked portal on Mesosphere DC/OS which is a Docker micro service
schedular/execution environment.
The whole thing is that we depend on a certain type of configuration layout
to launch these docker images... marathon definition files. With this we
are not able to launch a docker volume image type... so we need to find a
way to get the /static data generated by kickstarter into a Docker
container. But it seems that everything within this project is a micro
service and that it is not possible to build the portal-api as just one
Docker image. If this does exist.... PLEASE provide me the way because I
can't find it anywhere in the documentation... which I have read completely.

Also is there a ENV VAR that we can provide to not launch the portal api
service on loading of the default image? Just a question...

Kind regards,

Daniel

Daniel Vangrieken

Email : daniel.vangrieken@gmail.com
Phone: +32 (0) 476 / 99 65 61

2016-11-22 16:40 GMT+01:00 Martin Danielsson notifications@github.com:

Hey Daniel, yes, the source is this repository; I just realized I hadn't
recreated the documentation on readthedocs in a while, have to look into
automating that (I actually thought it would react to git changes, but
apparently it doesn't).

Running all of these things "natively" (i.e. not in docker) is currently
not actively supported, so there is usually no need to build your own
wicked containers (they're published to hub.docker.com).

Check out this on how to use the Kickstarter to create your configuration
repository:
https://github.com/Haufe-Lexware/wicked.haufe.io/blob/
master/doc/creating-a-portal-configuration.md

And this on how to create a docker-compose.yml file which usually works
out of the box for local deployments:
https://github.com/Haufe-Lexware/wicked.haufe.io/blob/
master/doc/deploying-locally.md

When you have that up and running, it's usually a small step to get it up
and running on an external docker host (it's just a question of which
docker-machine is active, and the IP/DNS and certificate configuration).

If you get stuck anyplace, feel free to file an issue here (like this
one). And in case of doubt, this repository is the source of truth.

Best regards,
Martin

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
#34 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AS4X3iJZk3c9tXze1esouFr0i11XQ4Euks5rAwzegaJpZM4K5VKY
.

[DonMartin76]

Hi David,

Unfortunately I'm not familiar with Mesos, but there has to be a means of storing persistent data for it. It doesn't matter much which means you choose, as long as the static data ends up at /var/portal-api/static, and the data in /var/portal-api/dynamic is persisted. There are multiple volumes involved:

• Static configuration data (currently built before deploying using docker-compose)
• Dynamic configuration data (a volume based on portal-api)
• Kong's PostgreSQL data volume (which is, depending on your demands, not required to be persistent)

This seems to be the way currently: https://mesosphere.github.io/marathon/docs/persistent-volumes.html or using "external volumes". It's not rocket science with the volumes at all, it's just one solution on how to persist data in your APIm configuration and dynamic data. But you won't get around using volumes one way or the other.

Can you use locally built docker images with Mesos? In that case you can, if you just copy your static configuration into a container image based on portal-api (which is normally the case anyway), you'd be just fine just running that thing instead of portal-api and ditch the volume one, as it's identical to the portal-api one.

Please report back how things are going and if you think I can help more.

BR, Martin

[DonMartin76]

FWIW, it seems I am running into similar problems when deploying to a Kubernetes Cluster.

[DonMartin76]

If you have a private repository you can use, then you can use the following Dockerfile inside the static directory:

FROM haufelexware/wicked.portal-api:<desired version tag>

COPY . /tmp/portal-api

RUN rm -rf /var/portal-api/static \
    && cp -R /tmp/portal-api /var/portal-api/static \
    && date +%s > /var/portal-api/static/.dockerbuild \
    && date -u "+%Y-%m-%d %H:%M:%S" > /var/portal-api/static/build_date

CMD ["npm", "start"]

Push the resulting image to your private repository, and use that instead of the haufelexware/wicked.portal-api:<desired version tag> image in your marathon definitions. I haven't tried it, but it should work.

You still have to have a volume (but an initially empty one) for /var/portal-api/dynamic and the data for PostgreSQL (see docker-compose.yml on where that has to be mounted into which container).

As soon as I have my Kubernetes thing running, I may be able to say more (and document it). If you have a Mesos installation of wicked running, I would happily incorporate hints on how to run wicked on Mesos into the documentation.

/Martin

[DonMartin76]

Will try out the following thing for use with Mesos/Kubernetes: If you supply GIT_CREDENTIALS (username:password) and GIT_REPO (and optionally GIT_REPO), the portal-api would check out the configuration from the supplied git repository using the given credentials.

Like this: git clone https://${GIT_CREDENTIALS}@${GIT_REPO}.

Would that solve your issue?

[Krieke]

Hello Martin, Sorry for the late reply. I got everything working on mesos but it will not allow us to make some automated build and deploying platform and this is ofcourse what we want. What I currently did was manually copy the static data folder on to a node in the cloud. In that node I manually launched a docker container with a mounted volume to the copied static folder. Then the docker is launched without errors. (during the launch of the docker I also mounted a rex-ray volume to a tmp dir). After the launch I moved the static folder onto the external mounted rex-ray volume. Then and only then we were able to launch the Portal Api via a marathon config file since it then could find the static data required to operate. It would be beter that one could build a docker image that would copy the static data folder created by kickstarter into the docker image itself. Like mentioned: this is a static folder so why would you create a separate mount volume for it. At that moment in time you could version your Dockerfile + static data content in GIT. In Docker cloud one create a new label/tag tag will be used within the marathon / cubernetes config file. This would make live so much easier for everyone. Don't you agree? PS: The pasword thing with GIT could create a security bridge to private repos. Kind regards, Daniel ================================================================ Daniel Vangrieken Email : daniel.vangrieken@gmail.com Phone: +32 (0) 476 / 99 65 61 2016-11-28 11:35 GMT+01:00 Martin Danielsson <notifications@github.com>:

…

Will try out the following thing for use with Mesos/Kubernetes: If you supply GIT_CREDENTIALS (username:password) and GIT_REPO (and optionally GIT_REPO), the portal-api would check out the configuration from the supplied git repository using the given credentials. Like this: git clone https://${GIT_CREDENTIALS}@${GIT_REPO}. Would that solve your issue? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#34 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AS4X3m0i7Rxip6xsJca0_Qi4SMBcZesVks5rCq5xgaJpZM4K5VKY> .

[DonMartin76]

The next update of the wicked Portal API will be able to at startup clone a specific version of a git repository into the container, without the need of creating a volume for it.

The "old" way of injecting a volume will still work, you just have to leave out the git credentials and repository.

If you feel like testing it out, you may (don't use for production!) pull the next tag of the portal API: haufelexware/wicked.portal-api:next. At startup, specify the env variable GIT_CREDENTIALS as username:password and GIT_REPO to e.g. bitbucket.org/yourorg/your.apim.config.git. Until release I will also add an option to retrieve a specific revision, right now it will clone master/HEAD.

[DonMartin76]

OK, so this will come in 0.11.0, currently working on the documentation. Would you mind sharing some information on how to run this on Mesos as soon as the 0.11.0 release is out?

[Krieke]

Hello Martin, Currently we make use of a private Docker images that has the static folder generated by "kickstarter" within its image. We didn't prefer to make use of the GIT pull stuff due to security issues. I have although a lot of other questions which I will add as an issue in your GIT Repository. Kind regards, Daniel ================================================================ Daniel Vangrieken Email : daniel.vangrieken@gmail.com Phone: +32 (0) 476 / 99 65 61 2016-12-09 9:46 GMT+01:00 Martin Danielsson <notifications@github.com>:

…

OK, so this will come in 0.11.0, currently working on the documentation. Would you mind sharing some information on how to run this on Mesos as soon as the 0.11.0 release is out? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#34 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AS4X3kBzxZ75dqWJUD9XU2V8BFsL6erKks5rGRVvgaJpZM4K5VKY> .

[DonMartin76]

Sure, go ahead. If you don't mind, I'd close this issue?

[Krieke]

OK