clean_results doesn't work with a large amount of results

[eilonreifs]

http://allure-srv:****/allure-docker-service/clean-results?project_id=__proj_id does not work when the results directory contains 44780 files.

Log output during the bug

allure_1     | Cleaning results for PROJECT_ID: __proj_id
allure_1     | /app/cleanAllureResults.sh: line 8: /bin/ls: Argument list too long
allure_1     | Results cleaned for PROJECT_ID: __proj_id

steps to reproduce:
send 50000 results, related to a project.
try to clean the results for this project.

suggestion:
if youre using rm * (which raises that exception)
you could replace it with find . -maxdepth 1 -type f -exec rm -r {} +

BTW - Thank you for this great tool, I'm using it for about 3 month without any bugs, and it was really simple to it set up.

[fescobar]

Hi @eilonreifs I used to use the command suggested by you a few months ago and we had the same problem when you have a lot of files in that directory. I'm using this script:

https://github.com/fescobar/allure-docker-service/blob/master/allure-docker-scripts/cleanAllureResults.sh#L8

 ls -d $PROJECT_RESULTS_DIRECTORY/* | grep -v history | xargs rm 2> /dev/null

I see the same problem. Can you create different projects to avoid this issue, please?

[eilonreifs]

Hi @fescobar,
find $PROJECT_RESULTS_DIRECTORY/ -maxdepth 1 -type f -exec rm -r {} + worked for me with 100000 files, when I put it in cleanAllureResults.sh instead of
ls -d $PROJECT_RESULTS_DIRECTORY/* | grep -v history | xargs rm 2> /dev/null, which didn't work

I assume its better to use 'find' anyway, what do you think?

BTW, what do you mean by creating different projects? for each run?

Thank you!

[fescobar]

@eilonreifs you could group your executions by project. Different suites using different project workspaces.
https://github.com/fescobar/allure-docker-service#creating-our-first-project

I need to test the command that you are suggesting. It will take time. I will put it in the backlog.
Thank you

[eilonreifs]

@fescobar
I already use multi projects, so It's not an option for me.
Do you have an estimation? I'll consider implementing a temp solution for the meantime

Thanks a lot! I really appreciate your contribution, and especially the fast responses.

[fescobar]

@eilonreifs probably the next year.

[fescobar]

@eilonreifs I will try to fix it earlier.
This issue is related to this one #58

[RafaelloLoliop]

Hey, I also encountered this problem and solve it using find

find $DIR -maxdepth 1 -print0 | xargs -0 rm
based on discussion from:
https://stackoverflow.com/questions/11289551/argument-list-too-long-error-for-rm-cp-mv-commands

Some help methods to generate sample:

max=1000000;i=1; while ((i<=max)); do touch $(seq $i $((i+99))); i=$((i+100)); done
mkdir history

1. Current

ls -d ./* | grep -v history | xargs rm 2> /dev/null
bash: /bin/ls: Argument list too long

ls -l | grep ^- | wc -l
1000000

2. New

find $DIR -maxdepth 1 -print0 | xargs -0 rm
rm: cannot remove '/home/tmp/file_tests': Is a directory
rm: cannot remove '/home/tmp/file_tests/history': Is a directory

ls -l | grep ^- | wc -l
0

I hope it will help a little and happy new year :)

[fescobar]

@RafaelloLoliop Thank you. I will try your improvement soon.

[RafaelloLoliop]

@fescobar
If u don't have time I can prepare PR with this change? But I'm not aware of others problems and also you mention that

Hi @eilonreifs I used to use the command suggested by you a few months ago and we had the same problem when you have a lot of files in that directory

in the past something also was wrong here.

[fescobar]

@RafaelloLoliop I'm working on multiple changes. You will have news this week or the next one.

[fescobar]

@eilonreifs @RafaelloLoliop can you check the fix with the current beta version?

image: "frankescobar/allure-docker-service:beta"

I don't know what is the limitation. But test it as much as you can.
Let me know when you finished testing.

Thank you.

[RafaelloLoliop]

@fescobar
Applied new version for my projects. Error was not very common (like once per month on ~20 projects) so cannot confirm that everything is okay atm, but I will monitor. Hope that it is working as expected.

Thanks

[fescobar]

@RafaelloLoliop I've tested using your bash file to create multiple users. I've created more than 40k files and I could reproduce the error. With the fix, the error disappears.

[fescobar]

@eilonreifs @RafaelloLoliop this fix was released from version 2.13.8:

https://github.com/fescobar/allure-docker-service/releases/tag/v2.13.8
https://github.com/fescobar/allure-docker-service-ui/releases/tag/v7.0.3

Thank you for your help.