Windows docker compose error: 'hdfs-namenode' failed to build

[re2023]

When running:
docker-compose up
It fails to build on on step 11 for hadoop with the following errors:
Building hdfs-namenode Step 1/29 : ARG BUILDER_IMAGE_NAME=debian Step 2/29 : ARG BUILDER_IMAGE_TAG=stretch-20200224-slim Step 3/29 : ARG BASE_IMAGE_NAME=debian Step 4/29 : ARG BASE_IMAGE_TAG=stretch-20200224-slim Step 5/29 : ARG HADOOP_VERSION=3.2.1 Step 6/29 : FROM ${BUILDER_IMAGE_NAME}:${BUILDER_IMAGE_TAG} as builder ---> 4e6990ebcef5 Step 7/29 : ARG HADOOP_VERSION ---> Using cache ---> c3425501241c Step 8/29 : ARG HADOOP_DOWNLOAD_URL=https://archive.apache.org/dist/hadoop/common/hadoop-${HADOOP_VERSION}/hadoop-${HADOOP_VERSION}.tar.gz ---> Using cache ---> 95feaf235d1d Step 9/29 : ARG HADOOP_APPLY_PATCHES=false ---> Using cache ---> 64fb2c4206d8 Step 10/29 : COPY ./files/ / ---> Using cache ---> b5bc9b6912bd Step 11/29 : RUN ./build-hadoop.sh ---> Running in 84dfe0ac80ec : invalid option ERROR: Service 'hdfs-namenode' failed to build : The command '/bin/sh -c ./build-hadoop.sh' returned a non-zero code: 1

[n3101]

@re2023 Apologies for the silence. I'll get someone to look into this.

[re2023]

Is there any update on this item?

[n3101]

@re2023 Oh no, another year gone by... I failed to get anyone to look at this. My really bad.

As a non-docker literate eng, I suspect that the root of your problem is:

Step 5/29 : ARG HADOOP_VERSION=3.2.1

The current Gaffer is tied to the obsolete Hadoop 2.6.5.

We have just released a Gaffer 2.0 alpha3 which allows a choice of the current Hadoop 2/Accumulo 1 (for legacy support) or Hadoop 3.3.3/Accumulo2.0.1. However, this alpha release is not in dockerhub yet. @t92549 @i471828 A couple of our engineers are working on this at the moment. I've asked them to look into your issue as part of that work and will get them to confirm my hypothesis. (Guys, please note:I see that the build-hadoop script suggests that a user should be able to supply their own hadoop version tarball, but I thought this had been withdrawn because of dependency issues?)

[t92549]

@n3101 gaffer-docker has always used hdfs 3.2.X. You can provide your own tar but that shouldn't be the cause of the issue.

@re2023 what platform are you running on? Building this on ubuntu works for me. Although, in the past I have had issues with this step where I needed to restart the docker service.

[n3101]

@t92549 told you I was not docker literate. OP apparently using Debian.

[t92549]

@t92549 told you I was not docker literate. OP apparently using Debian.

How do you know he is using Debian?

[n3101]

from the error msg:
ARG BUILDER_IMAGE_NAME=debian
ARG BASE_IMAGE_NAME=debian

[n3101]

@t92549 The only other thing I'd suggest is that this might be analogous to a problem I had when I tried to run this stuff - namely that things weren't started in the right order. In that case it was possible to use the docker desktop to start the missing service. This was just long enough ago for my chat with ctas to have been deleted.

[t92549]

debian is the base image from which we build the hdfs image. I was wondering what platform they are running docker on. Theoretically it shouldn't matter as long as it's Linux, I just haven't tested much else other than an ubuntu VM personally.

[n3101]

oh ok. I'll shut up now!

[re2023]

@n3101 this is running on docker running on windows. Apologies I am not very experienced in this area and just took the base docker images and tried to run them.

#run hdsf container
docker run -d --name hdfs -p 9870:9870 -v /data/hdfs:/data gchq/hdfs
#run accumulo container
docker run -d --name accumulo -p 9995:9995 --link hdfs:hdfs gchq/accumulo
# run gaffer-rest container
docker run -d --name gaffer-rest -p 8080:8080 gchq/gaffer-rest
# run gaffer-ui container
docker run -d --name gaffer-ui -p 5000:5000 gchq/gaffer-ui
#run gaffer container and link to resources
docker run -d --name gaffer --link hdfs:hdfs --link accumulo:accumulo --link gaffer-rest:gaffer-rest --link gaffer-ui:gaffer-ui gchq/gaffer

Due to my lack of experience I was hoping for more explicit guidance on how to get this running.

I have built a web app and want to connect to gaffer via the api, but am failing to get it running at this stage and not even sure if I am approaching this in the correct manner.

[t92549]

I don't have much experience using docker for Windows but it should include a VM under the hood so it can run Linux based containers.
If you want an accumulo based Gaffer like in the example you posted, this README and docker-compose file should explain how to do that.
If you just want to experiment with Gaffer running an in memory store rather than Accumulo you could run the ui instead.

Ultimately though I am not sure why you get that error message that fails to build the hdfs image, it may be a Windows thing.

[re2023]

I had tried that already running "docker-compose up" but that encounters the same error with hdfs

[re2023]

Here is the result:

[t92549]

It looks like the step that installs a bunch of packages is cancelled, which is why the build-hadoop.sh stage fails. After some googling it would seem that this can be caused by a firewall on Windows blocking docker.

If you don't mind testing this without the hdfs/Accumulo backing store, you could try just running the rest-api. This should just spin up the rest-api using an in memory MapStore rather than building hdfs and Accumulo.

[i471828]

@re2023 it would be interesting to see if the same error occurs if you are running the same steps inside a Linux VM - do you think you'd be able to try that? Maybe using VirtualBox or something similar, and a Ubuntu base image (or whatever Linux based system you prefer?

That would give a comparison to establish whether it's anything in the steps you're following, or whether it's a machine or OS specific issue.

[re2023]

It looks like the step that installs a bunch of packages is cancelled, which is why the build-hadoop.sh stage fails. After some googling it would seem that this can be caused by a firewall on Windows blocking docker.

If you don't mind testing this without the hdfs/Accumulo backing store, you could try just running the rest-api. This should just spin up the rest-api using an in memory MapStore rather than building hdfs and Accumulo.

I have added docker to allowed applications on local FW and it still fails.

However, you are correct that the gaffer-rest runs without issue.

@i471828 i will try spin up a linux vm and see if it experiences the same issue.

[re2023]

@i471828 you are correct. using a Linux VM in virtual box was able to execute the docker-compose up and once the network was bridged I could connect to the ui and rest.

I believe you are correct that this must be a windows related issue and not from your build.

I will close this. Using a Linux VM is a good workaround and more typical of a cloud environment where this would typically be hosted.

[t92549]

Good news you got it working! Apologies there is no documentation that states there might be issues with Windows builds

[n3101]

@t92549 @re2023 "Using a Linux VM is a good workaround and more typical of a cloud environment where this would typically be hosted."

We must document this.

[re2023]

Documenting that there is a compatibility issue with windows is good. but I do not feel you would need to spend any more effort on this.

I think it is really only applicable is someone was perhaps doing development work hosted a tried to host Gaffer locally.

I appreciate all your help, now to try and make use of Gaffer

[n3101]

@re2023 I'm not proposing a major doc effort for this, but we are currently redesigning our doc and identifying holes in it, so now is a good time for us to capture this before the knowledge is lost again.

I hope that things will go more easily for you now, but if not please contact us and we will try to respond more promptly.