Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Memory mapping with files in the old format (Klusters) #32

Open
rossant opened this issue Dec 5, 2013 · 4 comments
Open

Memory mapping with files in the old format (Klusters) #32

rossant opened this issue Dec 5, 2013 · 4 comments

Comments

@rossant
Copy link
Member

rossant commented Dec 5, 2013

There would be no more mandatory conversion before opening the files.

@nippoo
Copy link
Member

nippoo commented Dec 6, 2013

How would we deal with saving? Would we have to continue supporting this code?

@rossant
Copy link
Member Author

rossant commented Dec 6, 2013

Hmm. I can see three options:

  1. Full identical support of KWIK & Klusters (save multiple clusterings, cluster colors, groups, etc.). Complex since we need to define new file formats with these metadata.
  2. Supporting saving the clusters only (no colors, groups). Only the .clu file is modified.
  3. Read-only support: when loading files in the old format, the user would be prompted to either open the old files quickly in read-only mode, or to convert them to the new file format (slow conversion required first).

I'm leaning toward 3. I think 1 is too much work and not worth it, and 2 is not a good idea (you would want the cluster groups too, at least).

@kdharris101 what do you think? (the question was: whether to support the old file format in KlustaViewa without mandatory conversion, conversion which is a huge drawback according to many users who contacted me).

@kdharris101
Copy link
Member

Hi Cyrille,

I’m sorry I didn’t understand the context (or missed a previous email). Can you explain in more detail?

k

From: Cyrille Rossant [mailto:notifications@github.com]
Sent: 06 December 2013 09:11
To: klusta-team/klustaviewa
Cc: Harris, Kenneth
Subject: Re: [klustaviewa] Memory mapping with files in the old format (Klusters) (#32)

Hmm. I can see three options:

  1. Full identical support of KWIK & Klusters (save multiple clusterings, cluster colors, groups, etc.). Complex since we need to define new file formats with these metadata.
  2. Supporting saving the clusters only (no colors, groups). Only the .clu file is modified.
  3. Read-only support: when loading files in the old format, the user would be prompted to either open the old files quickly in read-only mode, or to convert them to the new file format (slow conversion required first).

I'm leaning toward 3. I think 1 is too much work and not worth it, and 2 is not a good idea (you would want the cluster groups too, at least).

@kdharris101https://github.com/kdharris101 what do you think? (the question was: whether to support the old file format in KlustaViewa without mandatory conversion, conversion which is a huge drawback according to many users who contacted me).


Reply to this email directly or view it on GitHubhttps://github.com//issues/32#issuecomment-29973469.

@rossant
Copy link
Member Author

rossant commented Dec 6, 2013

@kdharris101 I'd like to add the possibility to open files in the old file format without any conversion. I recently discovered a very simple technique in Python that allows to do that efficiently, both in terms of loading times and memory consumption. Many people are complaining about the need to convert their terabytes of data before being able to open them in KlustaViewa. I think people will have files in the old format for still quite a few years...

A possibility would be to only support a read-only mode for the old files.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants