Can't import files with non alphanumeric chars in path

[Nedfire2347]

Hello again, I'm currently working with @src7 on some dumps
And we can't import these kinds of sample with bin/import_dir.py because of their names.

Samples Examples :
Importing folder named : Collection #1_BTC combos
Import hieracrhi with files named : api_scrape_item.php?i=aZe0Rt1Y

Concerned code in this file :

AIL-framework/bin/import_dir.py

Line 70 in 1f8c858

for dirname, dirnames, filenames in os.walk(args.directory):

Notice : It could also lead to an exploitable vulnerabilty

[src7]

Hello,

For now I use this workaround to rename
to_import/1970/01/01/api_scrape_item.php?i=aZe0Rt1Y
to
to_import/1970/01/01/aZe0Rt1Y.txt
in batch in a very efficient way.

Command (in the to_import folder)
mmv ';*\=*' '#1#3.txt' (use -n to test before)

[Terrtia]

Hi @Nedfire2347 @src7 !
Thanks for the report !

The issue is related to the white-space in the path. (we use this separator in the Mixer module)

Fixed with 72fe8a2

[src7]

Hi,

what about the ? and the = ?

[Terrtia]

you right !
I removed some special characters with bdf2fce

[src7]

Nice

(it is a bit extreme but two files named &.txt and ?.txt can't coexist then ? Not a problem for me)

[src7]

First of all, there is a new dependency to add : python3-magic

But is this output normal ?
Some files are not gzipped ?

import_dir/pastebin>>to_import/2020/01/09/xqjKa4qa.txt.gz
import_dir/pastebin>>to_import/2020/01/09/xqShrUPm.txt
import_dir/pastebin>>to_import/2020/01/09/xqRx2kKG.txt.gz
import_dir/pastebin>>to_import/2020/01/09/xqDegaHQ.txt
import_dir/pastebin>>to_import/2020/01/09/xpvqC3p5.txt.gz
import_dir/pastebin>>to_import/2020/01/09/xpmfG0Fx.txt.gz
import_dir/pastebin>>to_import/2020/01/09/xpi1fgiw.txt.gz
import_dir/pastebin>>to_import/2020/01/09/xpDbbCTK.txt.gz
import_dir/pastebin>>to_import/2020/01/09/xnw2wdy2.txt
import_dir/pastebin>>to_import/2020/01/09/xnvEzU17.txt.gz

[Terrtia]

Thanks for the feedback !
All the files to import need to be gzipped.
I removed the python3-magic dependency (already installed in the requirement).
The importer use the magic number to check if a file is gzip compressed.

Fixed with 873797d

Files with the same file-path but different content are renamed by the Global Module(with an uuidv4)