Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

2.7 - Docker image broken #1201

Closed
agrantdeakin opened this issue Jul 26, 2021 · 8 comments · Fixed by #1204
Closed

2.7 - Docker image broken #1201

agrantdeakin opened this issue Jul 26, 2021 · 8 comments · Fixed by #1204
Assignees
@dadoonet
Labels
bug For confirmed bugs
Milestone
2.7

Comments

@agrantdeakin
Copy link

agrantdeakin commented Jul 26, 2021 •
edited

Describe the bug

The dadoonet/fscrawler 2.7 Docker images are broken.

E.g.

fscrawler        | Error: Unable to access jarfile /usr/share/fscrawler/lib/fscrawler-es7-2.7-SNAPSHOT.jar

Job Settings

N/A

Logs

fscrawler        | Error: Unable to access jarfile /usr/share/fscrawler/lib/fscrawler-es7-2.7-SNAPSHOT.jar

Expected behavior

Image should run without code 0/1.

Versions:

fscrawler-es7-2.7-SNAPSHOT.jar

Attachment

N/A
@agrantdeakin agrantdeakin added the check_for_bug Needs to be reproduced label Jul 26, 2021
@agrantdeakin agrantdeakin changed the title Docker image broken 2.7 - Docker image broken Jul 26, 2021
@agrantdeakin
Copy link
Author

The snapshot JAR names don't match what the image expects (have a date).

image

@dadoonet dadoonet added bug For confirmed bugs and removed check_for_bug Needs to be reproduced labels Jul 27, 2021
@dadoonet dadoonet self-assigned this Jul 27, 2021
@dadoonet
Copy link
Owner

Thanks for reporting. that's indeed a bug I can reproduce with:

docker pull dadoonet/fscrawler
docker run -it --rm -v ~/.fscrawler:/root/.fscrawler -v ~/tmp:/tmp/es fscrawler job_name

@dadoonet
Copy link
Owner

That's weird. The only jar which behaves like this is indeed fscrawler-es7-2.7-SNAPSHOT.jar:

docker run -it --rm dadoonet/fscrawler ls -l ./lib | grep fscrawler

gives

-rw-r--r-- 1 root root    18330 Jul 26 04:49 fscrawler-beans-2.7-SNAPSHOT.jar
-rw-r--r-- 1 root root    16350 Jul 26 04:49 fscrawler-cli-2.7-SNAPSHOT.jar
-rw-r--r-- 1 root root    26846 Jul 26 04:49 fscrawler-core-2.7-SNAPSHOT.jar
-rw-r--r-- 1 root root     4686 Jul 26 04:49 fscrawler-crawler-abstract-2.7-SNAPSHOT.jar
-rw-r--r-- 1 root root     4947 Jul 26 04:49 fscrawler-crawler-fs-2.7-SNAPSHOT.jar
-rw-r--r-- 1 root root     6282 Jul 26 04:49 fscrawler-crawler-ssh-2.7-SNAPSHOT.jar
-rw-r--r-- 1 root root    19283 Jul 26 04:49 fscrawler-elasticsearch-client-base-2.7-SNAPSHOT.jar
-rw-r--r-- 1 root root    22930 Jul 26 04:49 fscrawler-elasticsearch-client-v7-2.7-SNAPSHOT.jar
-rw-r--r-- 1 root root     4918 Jul 26 04:49 fscrawler-es7-2.7-20210726.044935-205.jar
-rw-r--r-- 1 root root    60497 Jul 26 04:48 fscrawler-framework-2.7-SNAPSHOT.jar
-rw-r--r-- 1 root root    15496 Jul 26 04:49 fscrawler-rest-2.7-SNAPSHOT.jar
-rw-r--r-- 1 root root    37201 Jul 26 04:49 fscrawler-settings-2.7-SNAPSHOT.jar
-rw-r--r-- 1 root root    13935 Jul 26 04:49 fscrawler-tika-2.7-SNAPSHOT.jar
-rw-r--r-- 1 root root    14324 Jul 26 04:49 fscrawler-workplacesearch-client-2.7-SNAPSHOT.jar

@dadoonet dadoonet added this to the 2.7 milestone Jul 27, 2021
@dadoonet
Copy link
Owner

Interesting. When I build it locally, the content looks good:

-rwxr-xr-x  0 dpilato staff    1435 20 jul 11:53 ./bin/fscrawler
-rw-r--r--  0 dpilato staff    1740 20 jul 11:53 ./bin/fscrawler.bat
-rw-r--r--  0 dpilato staff   22930 27 jul 12:38 ./lib/fscrawler-elasticsearch-client-v7-2.7-SNAPSHOT.jar
-rw-r--r--  0 dpilato staff   19283 27 jul 12:38 ./lib/fscrawler-elasticsearch-client-base-2.7-SNAPSHOT.jar
-rw-r--r--  0 dpilato staff   18330 27 jul 12:38 ./lib/fscrawler-beans-2.7-SNAPSHOT.jar
-rw-r--r--  0 dpilato staff   60517 27 jul 12:37 ./lib/fscrawler-framework-2.7-SNAPSHOT.jar
-rw-r--r--  0 dpilato staff   37201 27 jul 12:37 ./lib/fscrawler-settings-2.7-SNAPSHOT.jar
-rw-r--r--  0 dpilato staff    14324 27 jul 12:38 ./lib/fscrawler-workplacesearch-client-2.7-SNAPSHOT.jar
-rw-r--r--  0 dpilato staff    16350 27 jul 12:38 ./lib/fscrawler-cli-2.7-SNAPSHOT.jar
-rw-r--r--  0 dpilato staff    26846 27 jul 12:38 ./lib/fscrawler-core-2.7-SNAPSHOT.jar
-rw-r--r--  0 dpilato staff    13935 27 jul 12:38 ./lib/fscrawler-tika-2.7-SNAPSHOT.jar
-rw-r--r--  0 dpilato staff     4947 27 jul 12:38 ./lib/fscrawler-crawler-fs-2.7-SNAPSHOT.jar
-rw-r--r--  0 dpilato staff     4686 27 jul 12:38 ./lib/fscrawler-crawler-abstract-2.7-SNAPSHOT.jar
-rw-r--r--  0 dpilato staff     6282 27 jul 12:38 ./lib/fscrawler-crawler-ssh-2.7-SNAPSHOT.jar
-rw-r--r--  0 dpilato staff    15496 27 jul 12:38 ./lib/fscrawler-rest-2.7-SNAPSHOT.jar
-rw-r--r--  0 dpilato staff     4918 27 jul 12:38 ./lib/fscrawler-es7-2.7-SNAPSHOT.jar

@dadoonet
Copy link
Owner

So the problem is happening when running the deploy phase apparently:

[INFO] Reading assembly descriptor: /home/runner/work/fscrawler/fscrawler/distribution/es7/../src/main/assembly/assembly.xml
Downloading from oss-snapshots: https://oss.sonatype.org/content/repositories/snapshots/fr/pilato/elasticsearch/crawler/fscrawler-es7/2.7-SNAPSHOT/fscrawler-es7-2.7-20210726.044935-205.pom
Progress (1): 1.4 kB
                    
Downloaded from oss-snapshots: https://oss.sonatype.org/content/repositories/snapshots/fr/pilato/elasticsearch/crawler/fscrawler-es7/2.7-SNAPSHOT/fscrawler-es7-2.7-20210726.044935-205.pom (1.4 kB at 50 kB/s)
[INFO] Building tar: /home/runner/work/fscrawler/fscrawler/distribution/es7/target/docker/***/fscrawler/2.7-SNAPSHOT-es7-nolang/build/maven.tgz

It looks like we are downloading the latest SNAPSHOT that we just built locally before packaging the tar for docker...

I'm not an expert of that part so it might take some time to understand how to fix that in a proper way.

@dadoonet
Copy link
Owner

I can reproduce the behavior locally so it's not related to automated deploy from github actions 👍🏼

dadoonet added a commit that referenced this issue Jul 27, 2021
@dadoonet
Split build and deploy phases for Docker images
29a009d 
If we build the docker image in the deploy phase, something unexpected is happening.

The call to the deploy plugin, changes the name of the artifact when using a SNAPSHOT version.
That artifact is then used by the docker plugin and pulled from Sonatype **without** changing its name.
So we end up with a name like `fscrawler-es7-2.7-20210726.044935-205.jar` instead of the `finalName` which is `fscrawler-es7-2.7-SNAPSHOT.jar`.

The fscrawler job can not start because it calls:

```sh
exec "$JAVA" ${JAVA_OPTS} -cp "$FS_CLASSPATH" -jar "$FS_HOME/lib/${project.build.finalName}.jar" "$@"
```

The jar is then not found.

If we split the docker build and the docker deploy within 2 different phases, the build generates the right artifact name locally and the deploy "just" deploy what has already been generated.

This commit adds this behavior.

Closes #1201.
@dadoonet dadoonet mentioned this issue Jul 27, 2021
Merged
@dadoonet
Copy link
Owner

I think I found the way to fix it. Let's wait for Github actions to deploy the new version and see if everything is fine now...

@agrantdeakin
Copy link
Author

Thanks for the prompt fix.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@dadoonet dadoonet

Labels
bug For confirmed bugs
Projects
None yet
Milestone
2.7
Development

Successfully merging a pull request may close this issue.

Split build and deploy phases for Docker images dadoonet/fscrawler
2 participants
@dadoonet @agrantdeakin