Saving files with long names.

[ignace-computing]

Hello!

When you have a lot of simulation parameters, the filename under which you want to save your data mighty become too long. Here is a minimal working example of such a situation:

long_word = "thisIsAVeryLongWordAndItCanBeEvenLonger"
very_long_dict = Dict(:long_word => long_word, :a=>0.123)
very_long_dict[:other_long_word] = long_word*long_word

dict_with_results = Dict(:niceresults => true)
DATADIR_NAME = "testDir"

wsave(datadir(DATADIR_NAME, savename(very_long_dict, "bson")), dict_with_results)

That gives the following error:
IOError: stat: name too long (ENAMETOOLONG) for file ...

Would you have any ideas?
Thank you.

[Datseris]

Hi!

Limits of filenames come from the operating system, so my only suggestion is "reduce name size".

(this is also why this question is unrelated to DrWatson)

[ignace-computing]

Thank you for the response.
I understand why you closed the issue and also why you think it is unrelated.

However, I must say that, in real life, simulations often simply contain a lot of input parameters. It would be nice if DrWatson had an extra functionality, where it automatically maps a long file name onto a shorter one (and of course also does the bookkeeping, probably with a new [bidirectionally searchable] data structure mapping short names to longer ones).

So, yes this is an operating system issue, but I think it would be nice if DrWatson provides us with robust tools to circumvent it. Any ideas on how this could be implemented?

All the best,

[tamasgal]

I think the root of the problem is to squeeze a lot of information into the filename. I personally do not like putting information into the filename but rather use a local database or table to find meta data for my files. For simple things where you scan a parameter space for a few variables, the savename-approach is perfectly fine, but for anything which goes beyond that (and your example certainly does), you should think about a proper meta-data management.

That being said, if you have a use-case and you think you can generalise it, we are happy to discuss it in a PR! :)

[Datseris]

It would be nice if DrWatson had an extra functionality, where it automatically maps a long file name onto a shorter one (and of course also does the bookkeeping, probably with a new [bidirectionally searchable] data structure mapping short names to longer ones).

That's an interesting suggestion. I've closed this issue because you haven't said any of this beforehand, and I simply consider the issue an incorrect bug report, I didn't mean to disregard the suggestion.

However, I agree with Tamas' statement here. It is probably more useful to use a proper database, otherwise things might get out of hand quickly... DrWatson is not a data-management system (because e.g. we're getting CaosDB.jl soon in Julia), which is why I'd feel reluctant adding database-management-functionality here.

[omerchiers]

I think the root of the problem is to squeeze a lot of information into the filename. I personally do not like putting information into the filename but rather use a local database or table to find meta data for my files. For simple things where you scan a parameter space for a few variables, the savename-approach is perfectly fine, but for anything which goes beyond that (and your example certainly does), you should think about a proper meta-data management.

Probably it's not the best place to discuss this, but I would be very interested in seeing such a metadata management workflow. Maybe we could move that part of the discussion to discourse?

[Datseris]

Yes, sounds like the most reasonable.