Skip to content
This repository has been archived by the owner on Nov 17, 2023. It is now read-only.

fix the Scala URL issue for downloading data in scripts #12913

Merged
merged 1 commit into from
Oct 23, 2018
Merged

fix the Scala URL issue for downloading data in scripts #12913

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion scala-package/examples/scripts/infer/imageclassifier/get_resnet_18_data.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,5 +37,5 @@ if [ ! -f "$data_path" ]; then
wget https://s3.us-east-2.amazonaws.com/scala-infer-models/resnet-18/resnet-18-symbol.json -P $data_path
wget https://s3.us-east-2.amazonaws.com/scala-infer-models/resnet-18/resnet-18-0000.params -P $data_path
wget https://s3.us-east-2.amazonaws.com/scala-infer-models/resnet-18/synset.txt -P $data_path
wget https://s3.amazonaws.com/model-server/inputs/kitten.jpg -P $image_path
wget https://s3.us-east-2.amazonaws.com/mxnet-scala/scala-example-ci/resnet152/kitten.jpg -P $image_path
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe you are using S3 bucket to keep the model artifacts.
But, should the bucket name be mxnet-scala or something else ?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@piyushghai Previously when I set up with the CI, I created a bucket namely mxnet-scala. There are already mxnet-model or mxnet-modelsin the S3 bucket. I don't want to add additional folder inside to change something there.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My point is, if tomorrow there was a similar example for another language binding, say Java, then they would either have to :

  1. Create a new S3 bucket and host the same model files there.
    OR
  2. Point to an already existing centralized bucket.
    I don't think it's a scalable to follow point 1, which we seem to be doing here, because then we will end up having to duplicate hosting of files at other places as well.

There are also other cases, where let's say one wants to replace a trained model with a new state of the art trained model, the fewer the number of places in which this change happens, the easier it will be right ?

Copy link
Member Author

@lanking520 lanking520 Oct 22, 2018 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My point is, if tomorrow there was a similar example for another language binding, say Java, then they would either have to :

  1. Create a new S3 bucket and host the same model files there.
    OR
  2. Point to an already existing centralized bucket.
    I don't think it's a scalable to follow point 1, which we seem to be doing here, because then we will end up having to duplicate hosting of files at other places as well.

There are also other cases, where let's say one wants to replace a trained model with a new state of the art trained model, the fewer the number of places in which this change happens, the easier it will be right ?

But the fact is that some of the models with the same name cannot be used in Scala/Java. Such as Res152. Not all res152 models from MXNet Python can be successfully ported into MXNet Scala/Java especially those in data.mxnet.io. The cause behind it can be some diff on the precision (Float64) or input-type diffs. In that case, there has to be a Scala/Java model zoo somewhere to holds the code. Since Java is the child came from Scala, it should be fine for it to use the model directly from Scala.

I think model-server they have their own model zoo is also with the same reason, or they can port all models from one place directly.

Copy link
Contributor

@piyushghai piyushghai Oct 22, 2018 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see your point there. But we might not be using the best practice here.
I'll [reluctantly] approve this PR since the break is a critical break and fixes a bunch of malformed urls in examples

Copy link
Member

@nswamy nswamy Oct 23, 2018 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

its ok for now, I agree with @piyushghai that we need to make sure the same models work across all language bindings, otherwise it is confusing to users why the models are different and could lose their trust.

fi
8 changes: 4 additions & 4 deletions scala-package/examples/scripts/infer/imageclassifier/get_resnet_data.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,8 +34,8 @@ if [ ! -d "$image_path" ]; then
fi

if [ ! -f "$data_path" ]; then
wget http://data.mxnet.io/models/imagenet-11k/resnet-152/resnet-152-0000.params -P $data_path
wget http://data.mxnet.io/models/imagenet-11k/resnet-152/resnet-152-symbol.json -P $data_path
wget http://data.mxnet.io/models/imagenet-11k/synset.txt -P $data_path
wget https://s3.amazonaws.com/model-server/inputs/kitten.jpg -P $image_path
wget https://s3.us-east-2.amazonaws.com/mxnet-scala/scala-example-ci/resnet152/resnet-152-0000.params -P $data_path
wget https://s3.us-east-2.amazonaws.com/mxnet-scala/scala-example-ci/resnet152/resnet-152-symbol.json -P $data_path
wget https://s3.us-east-2.amazonaws.com/mxnet-scala/scala-example-ci/resnet152/synset.txt -P $data_path
wget https://s3.us-east-2.amazonaws.com/mxnet-scala/scala-example-ci/resnet152/kitten.jpg -P $image_path
fi
2 changes: 1 addition & 1 deletion scala-package/examples/scripts/infer/objectdetector/get_ssd_data.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ fi
if [ ! -f "$data_path" ]; then
wget https://s3.amazonaws.com/model-server/models/resnet50_ssd/resnet50_ssd_model-symbol.json -P $data_path
wget https://s3.amazonaws.com/model-server/models/resnet50_ssd/resnet50_ssd_model-0000.params -P $data_path
wget https://raw.githubusercontent.com/awslabs/mxnet-model-server/master/examples/ssd/synset.txt -P $data_path
wget https://s3.amazonaws.com/model-server/models/resnet50_ssd/synset.txt -P $data_path
cd $image_path
wget https://cloud.githubusercontent.com/assets/3307514/20012566/cbb53c76-a27d-11e6-9aaa-91939c9a1cd5.jpg -O 000001.jpg
wget https://cloud.githubusercontent.com/assets/3307514/20012567/cbb60336-a27d-11e6-93ff-cbc3f09f5c9e.jpg -O dog.jpg
Expand Down