Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

perf(tools): fabric 1.x AIO image pre-fetching #649

Closed
petermetz opened this issue Mar 10, 2021 · 0 comments · Fixed by #826
Closed

perf(tools): fabric 1.x AIO image pre-fetching #649

petermetz opened this issue Mar 10, 2021 · 0 comments · Fixed by #826
Labels
dependencies Pull requests that update a dependency file Developer_Experience enhancement New feature or request Fabric good-first-issue Good for newcomers Hacktoberfest Hacktoberfest participants are welcome to take a stab at issues marked with this label. Nice-to-Have Performance Everything related to how fast/efficient the software or it's tooling (e.g. build) is.

Comments

@petermetz
Copy link
Member

Description

Make the 1.x Fabric AIO image pre-fetch the docker images for Fabric at (AIO) image build time the same way the 2.x already does it here:

tools/docker/fabric-all-in-one/Dockerfile_v2.x

# jq is needed by the /download-frozen-image-v2.sh script to pre-fetch docker images without docker.
RUN apk add --no-cache jq
# Get the utility script that can pre-fetch the Fabric docker images without
# a functioning Docker daemon available which we do not have at image build
# time so have to resort to manually get the Fabric images insteadd of just saying
# "docker pull hyperledger/fabric..." etc.
# The reason to jump trough these hoops is to speed up the boot time of the
# container which won't have to download the images at container startup since
# they'll have been cached already at build time.
RUN curl -sSL https://raw.githubusercontent.com/moby/moby/dedf8528a51c6db40686ed6676e9486d1ed5f9c0/contrib/download-frozen-image-v2.sh > /download-frozen-image-v2.sh
RUN chmod +x /download-frozen-image-v2.sh
RUN mkdir -p /etc/hyperledger/fabric
RUN /download-frozen-image-v2.sh /etc/hyperledger/fabric/ hyperledger/fabric-peer:${FABRIC_VERSION}
RUN /download-frozen-image-v2.sh /etc/hyperledger/fabric/ hyperledger/fabric-orderer:${FABRIC_VERSION}
RUN /download-frozen-image-v2.sh /etc/hyperledger/fabric/ hyperledger/fabric-ccenv:${FABRIC_VERSION}
RUN /download-frozen-image-v2.sh /etc/hyperledger/fabric/ hyperledger/fabric-tools:${FABRIC_VERSION}
RUN /download-frozen-image-v2.sh /etc/hyperledger/fabric/ hyperledger/fabric-baseos:${FABRIC_VERSION}
RUN /download-frozen-image-v2.sh /etc/hyperledger/fabric/ hyperledger/fabric-ca:${CA_VERSION}

And in the network runner script of the container:

tar -cC '/etc/hyperledger/fabric/' . | docker load

#!/bin/sh
set -e

# Needed so that we have the "peer" binary on our path
export PATH=/fabric-samples/bin/:$PATH

# Source the utility that we use to parse semantic version strings in bash
. /semver.sh

function main()
{

  local MAJOR=0
  local MINOR=0
  local PATCH=0
  local SPECIAL=""
  semverParseInto "${FABRIC_VERSION}" MAJOR MINOR PATCH SPECIAL

  tar -cC '/etc/hyperledger/fabric/' . | docker load

  /bootstrap.sh ${FABRIC_VERSION} ${CA_VERSION} -b -s

  echo "[FabricAIO] >>> Parsed MAJOR version of Fabric as ${MAJOR}"

  if [ "$MAJOR" -gt 1 ]; then
    # Major version is 2 or newer (we'll deal with 3.x when it is released)
    cd /fabric-samples/test-network/
    echo "[FabricAIO] >>> pulling up test network..."
    ./network.sh up -ca
    echo "[FabricAIO] >>> test network pulled up OK."
    ./network.sh createChannel -c mychannel
    echo "[FabricAIO] >>> channel created OK."
    ./network.sh deployCC -ccn basic -ccp ../asset-transfer-basic/chaincode-go -ccl go
    echo "[FabricAIO] >>> contract deployed OK."
    echo "[FabricAIO] >>> container healthcheck should begin passing in about 5-15 seconds..."
  else
    # Major version is 1.x or earlier (assumption is 1.4.x only)
    cd /fabric-samples/fabcar/
    ./startFabric.sh
  fi

}

main

(add more description and relevant links)

Acceptance Criteria

  1. Tests are passing
  2. Fabric 1.x tests are sped up successfully due to the AIO container booting much faster by not having to download everything at container start from docker hub (the images inside the container for pulling up the actual ledger)

cc: @takeutak @sfuji822 @hartm @jonathan-m-hamilton @AzaharaC @jordigiam @kikoncuo

@petermetz petermetz added dependencies Pull requests that update a dependency file Developer_Experience enhancement New feature or request Fabric good-first-issue Good for newcomers Hacktoberfest Hacktoberfest participants are welcome to take a stab at issues marked with this label. Nice-to-Have Performance Everything related to how fast/efficient the software or it's tooling (e.g. build) is. labels Mar 10, 2021
petermetz added a commit to petermetz/cacti that referenced this issue Apr 14, 2021
Fixes hyperledger#649

In addition to that fix it also fixes a pre-fetching bug in the 2.x
image which was stopping it from actually using the pre-fetched images
on the file-system => the directory paths had to be specified
one by one for each image that we pre-fetched, it did not
work if you specified a common parent directory where the
images were stored, so this commit provides a fix for both
1.4.x and 2.x in the sense that they both should no longer
hit DockerHub for image downloads at container startup
time, only build-time as long as the FABRIC_VERSION
env vars were not altered by the person launching the
containers (if they were then those different versions
will still have to be fetched at runtime).

The biggest net positive effect(s) of this commit:
1. Image boot will be much faster, especially on machines
where the network is slow.
2. The risk of getting slapped with DockerHub image pull
rate limiting goes down significantly.

Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com>
petermetz added a commit to petermetz/cacti that referenced this issue Apr 14, 2021
Fixes hyperledger#649

In addition to that fix it also fixes a pre-fetching bug in the 2.x
image which was stopping it from actually using the pre-fetched images
on the file-system => the directory paths had to be specified
one by one for each image that we pre-fetched, it did not
work if you specified a common parent directory where the
images were stored, so this commit provides a fix for both
1.4.x and 2.x in the sense that they both should no longer
hit DockerHub for image downloads at container startup
time, only build-time as long as the FABRIC_VERSION
env vars were not altered by the person launching the
containers (if they were then those different versions
will still have to be fetched at runtime).

The biggest net positive effect(s) of this commit:
1. Image boot will be much faster, especially on machines
where the network is slow.
2. The risk of getting slapped with DockerHub image pull
rate limiting goes down significantly.

Also fixes a health check bug where the FABRIC_VERSION build arg
was not being exposed as an ENV var
in the container and the parsing of the
version was failing due to this.

Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com>
petermetz added a commit to petermetz/cacti that referenced this issue Apr 20, 2021
Fixes hyperledger#649

In addition to that fix it also fixes a pre-fetching bug in the 2.x
image which was stopping it from actually using the pre-fetched images
on the file-system => the directory paths had to be specified
one by one for each image that we pre-fetched, it did not
work if you specified a common parent directory where the
images were stored, so this commit provides a fix for both
1.4.x and 2.x in the sense that they both should no longer
hit DockerHub for image downloads at container startup
time, only build-time as long as the FABRIC_VERSION
env vars were not altered by the person launching the
containers (if they were then those different versions
will still have to be fetched at runtime).

The biggest net positive effect(s) of this commit:
1. Image boot will be much faster, especially on machines
where the network is slow.
2. The risk of getting slapped with DockerHub image pull
rate limiting goes down significantly.

Also fixes a health check bug where the FABRIC_VERSION build arg
was not being exposed as an ENV var
in the container and the parsing of the
version was failing due to this.

Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com>
petermetz added a commit to petermetz/cacti that referenced this issue Apr 20, 2021
Fixes hyperledger#649

In addition to that fix it also fixes a pre-fetching bug in the 2.x
image which was stopping it from actually using the pre-fetched images
on the file-system => the directory paths had to be specified
one by one for each image that we pre-fetched, it did not
work if you specified a common parent directory where the
images were stored, so this commit provides a fix for both
1.4.x and 2.x in the sense that they both should no longer
hit DockerHub for image downloads at container startup
time, only build-time as long as the FABRIC_VERSION
env vars were not altered by the person launching the
containers (if they were then those different versions
will still have to be fetched at runtime).

The biggest net positive effect(s) of this commit:
1. Image boot will be much faster, especially on machines
where the network is slow.
2. The risk of getting slapped with DockerHub image pull
rate limiting goes down significantly.

Also fixes a health check bug where the FABRIC_VERSION build arg
was not being exposed as an ENV var
in the container and the parsing of the
version was failing due to this.

Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com>
petermetz added a commit that referenced this issue Apr 21, 2021
Fixes #649

In addition to that fix it also fixes a pre-fetching bug in the 2.x
image which was stopping it from actually using the pre-fetched images
on the file-system => the directory paths had to be specified
one by one for each image that we pre-fetched, it did not
work if you specified a common parent directory where the
images were stored, so this commit provides a fix for both
1.4.x and 2.x in the sense that they both should no longer
hit DockerHub for image downloads at container startup
time, only build-time as long as the FABRIC_VERSION
env vars were not altered by the person launching the
containers (if they were then those different versions
will still have to be fetched at runtime).

The biggest net positive effect(s) of this commit:
1. Image boot will be much faster, especially on machines
where the network is slow.
2. The risk of getting slapped with DockerHub image pull
rate limiting goes down significantly.

Also fixes a health check bug where the FABRIC_VERSION build arg
was not being exposed as an ENV var
in the container and the parsing of the
version was failing due to this.

Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
dependencies Pull requests that update a dependency file Developer_Experience enhancement New feature or request Fabric good-first-issue Good for newcomers Hacktoberfest Hacktoberfest participants are welcome to take a stab at issues marked with this label. Nice-to-Have Performance Everything related to how fast/efficient the software or it's tooling (e.g. build) is.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant