Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Improve build script of examples #214

Merged
merged 1 commit into from
Oct 28, 2021
Merged

Improve build script of examples #214

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
80 changes: 73 additions & 7 deletions examples/build_image.sh
100644 → 100755
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,25 +14,91 @@
# See the License for the specific language governing permissions and
# limitations under the License.


# Reset in case getopts has been used previously in the shell
OPTIND=1

usage()
{
echo ""
echo "Usage: $0 -r repository dir_1 ... dir_n"
echo -e "\t-r The repository parameters allows to select a private Docker repository to upload the images to."
echo -e "\tThe script expects a list of Sedna example to build (joint_inference, federated_learning, etc..).
\tMultiple example can be built at the same time by passing a list of directories such as: dir_1 dir_2 ...
\tIf no directory is specified, the script will automatically build all available examples."
exit 1 # Exit script after printing help
}

while getopts "r:" opt
do
case "$opt" in
r ) IMAGE_REPO="$OPTARG" ;;
? ) usage ;; # Print usage in case parameter is non-existent
esac
done

shift $((OPTIND-1))

[ "${1:-}" = "--" ] && shift

type=$@

if [ -z "$type" ]
then
echo "No example directory/s specified, building all example images.."
type="all"
fi

if [ -z "$IMAGE_REPO" ]
then
echo "Using default Docker hub"
IMAGE_REPO="kubeedge"
fi

cd "$(dirname "${BASH_SOURCE[0]}")"

IMAGE_REPO=${IMAGE_REPO:-kubeedge}
IMAGE_TAG=${IMAGE_TAG:-v0.4.0}

EXAMPLE_REPO_PREFIX=${IMAGE_REPO}/sedna-example-

dockerfiles=(
dockerfiles_federated_learning=(
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually it can be improved by:

  • remove explict Dockerfiles here, using like ls *.Dockerfile, and then match by the subdirectories
  • Or move these Dockerfiles into their subdirectories

It can mitigate these explict Dockerfiles.

But because these dockerfiles are probably stable, it's optional, you can adopt if you want.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could maybe apply this change later. It depends on the example folder structure and if we change it we might lose backward compatibility.

Copy link

@JimmyYang20 JimmyYang20 Oct 27, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@vcozzolino there are two examples of federated learning. If the user only want to use the one, the other shouldn't be mirrored.

Copy link
Contributor Author

@vcozzolino vcozzolino Oct 27, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is more of a problem with the folder structure rather than the script. As @llhuii suggested, we use the directories name to build the associated Dockerfiles. If we want to do more specific pattern matching, we have a few options:

  1. Create two different directories for the two examples of federated learning such as federated_learning_mistnet and federated_learning_surface. It's anyway something you do in the nested directories. It's just a matter of moving up the splitting in the directory tree. This probably requires extra work to check that all dependencies and Dockerfiles are correctly updated.
  2. Move Dockerfiles into the subfolders. This breaks backward compatibility because the Dockerfiles use relative imports, so we will also have to change the content of each Dockerfile itself.
  3. Use Dockerfiles names to do the matching in the script rather than folder names. However, this might introduce ambiguity in the decision of which Dockerfiles to build.

Do you have suggestions?

Maybe option 3 is the least invasive.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

option 2 is higher cohesion, lower coupling. we can do it in future.

But for now the feature level is enough I think.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds good. So I guess we can go ahead with the current changes and plan for extra improvements in the future.

federated-learning-mistnet-yolo-aggregator.Dockerfile
federated-learning-mistnet-yolo-client.Dockerfile
federated-learning-surface-defect-detection-aggregation.Dockerfile
federated-learning-surface-defect-detection-train.Dockerfile
incremental-learning-helmet-detection.Dockerfile
)

dockerfiles_joint_inference=(
joint-inference-helmet-detection-big.Dockerfile
joint-inference-helmet-detection-little.Dockerfile
)

dockerfiles_lifelong_learning=(
lifelong-learning-atcii-classifier.Dockerfile
)

for dockerfile in ${dockerfiles[@]}; do
example_name=${dockerfile/.Dockerfile}
docker build -f $dockerfile -t ${EXAMPLE_REPO_PREFIX}${example_name}:${IMAGE_TAG} --label sedna=examples ..
dockerfiles_incremental_learning=(
vcozzolino marked this conversation as resolved.
Show resolved Hide resolved
incremental-learning-helmet-detection.Dockerfile
)

# Iterate over the input folders and build them sequentially.
for tp in ${type[@]}; do
if [[ "$tp" == "all" ]]; then
dockerfiles+=(
"${dockerfiles_federated_learning[@]}"
"${dockerfiles_joint_inference[@]}"
"${dockerfiles_lifelong_learning[@]}"
"${dockerfiles_incremental_learning[@]}")
else
dfiles=dockerfiles_$tp[@]
dockerfiles+=("${!dfiles}")
fi
done

# Removing duplicate entries (if any)
dockerfiles=($(echo "${dockerfiles[@]}" | tr ' ' '\n' | sort -u))

for dockerfile in ${dockerfiles[@]}; do
echo "Building $dockerfile"
example_name=${dockerfile/.Dockerfile}
docker build -f $dockerfile -t ${IMAGE_REPO}/${example_name}:${IMAGE_TAG} --label sedna=examples ..
done