Repository navigation
NDOS 0.1.0 is ready for pilot testing — looking for labs and collaborators #21
Elnazkarami
announced in
Announcements
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
A drive arrives. The person who filled it has graduated. Somewhere in it are three years of recordings, and nobody left can say with confidence which animal is which, what was injected when, or which of four folders called
finalproduced the published figures.That is the problem I built NDOS to address—and I am now looking for researchers and labs willing to test it on real data structures.
What NDOS does
NDOS is an open-source system for recovering, organizing, querying and tracing pre-publication animal-neuroscience data.
It reads an existing drive without changing the source data, and every value it reports records how it is known:
Structural inference is labelled too, but as a different kind of claim: every proposed placement carries the rule that produced it and how well the folder matched.
It then proposes a working view of the subjects, sessions and acquisitions represented by those files. Researchers can add experimental facts that cannot be observed from storage—such as surgeries, injections and interventions—without confusing declared metadata with machine inference.
Three parts are especially important.
Recovery
NDOS inventories heterogeneous storage, identifies duplicates and questionable files, and reads ZIP indexes—and, when requested, TAR archives—without extracting them.
It can propose a more understandable project layout while showing the rule and evidence behind every placement. Incorrect guesses can therefore be identified and corrected rather than silently trusted.
Evidence-aware discovery
Cohort queries return three outcomes:
A session whose species was never recorded is not a session known not to be a mouse. Treating those situations as equivalent can quietly bias a cohort toward whichever animals happened to have more complete metadata.
NDOS preserves that distinction and shows the evidence behind every query result. This is the part I would particularly like researchers to challenge, test and help improve.
Traceability
NDOS can record analysis runs and connect declared outputs to their commands, code version and source inputs. That makes it possible to trace a result or figure back through the recorded processing history to the files that supported it.
It also works for new projects
NDOS is not limited to inherited archives.
If you are beginning a project, it can create the project structure and generate folders for new recording sessions with the naming convention already applied:
The goal is to avoid creating the unreadable drive described in the first paragraph.
Why I built it
NDOS is an independent open-source project. I built the first release largely on my own, because I have repeatedly seen how much scientific time is lost when researchers cannot confidently navigate, interpret or reuse their own laboratory data.
The motivation is larger than file organization.
I want to help laboratories preserve the experimental meaning of their data, improve collaboration between researchers and create a more consistent language for subjects, sessions, procedures, acquisitions, evidence and provenance. Better internal organization should also make eventual validation, sharing and open-science participation more realistic for smaller laboratories—not only for groups with dedicated data-management teams.
But a system intended to work across laboratories cannot be shaped by one person's assumptions or the conventions of one lab.
That is why I do not want NDOS to remain a solo project.
I would be very happy to hear from:
You do not need to contribute code. A folder structure NDOS interprets incorrectly may be more valuable at this stage than a new feature.
Install and try it
Python 3.9 or newer is required, with no third-party runtime dependencies.
If you prefer an interface:
The interface runs locally, binds only to the loopback address and uses a per-launch token together with Host and Origin checks. Nothing is uploaded or exposed as a network service.
The complete walkthrough is in QUICKSTART.md and takes approximately fifteen minutes.
Safety and privacy
Inspection is read-only. Operations that can move, extract, delete or change permissions require an explicit option, show what they will do and ask for confirmation.
The default reconstructed layout uses symbolic links where supported, so the original files remain in place.
No file contents need to be shared when reporting feedback. Structural reports can contain paths or example filenames, and these can be anonymized if they are sensitive.
How it fits with existing workflows
The repository includes a draft organizational specification covering project structure, sessions, identifiers and data-status flags. That specification is one component of NDOS, not a requirement for using every tool.
If your lab already has a folder convention—or already uses something like NeuroBlueprint—you can still use the inventory, archive inspection, metadata, query and provenance functionality. NDOS is intended to complement that kind of specification rather than compete with it: much of its work happens before a layout can be applied at all, on data that was never organized to any convention.
NDOS can also produce an explicitly labelled BIDS-shaped export and an NWB conversion plan for NeuroConv. It does not claim validation or conformance that it has not demonstrated, and it does not attempt to replace established data formats or conversion tools.
What I am asking for now
NDOS 0.1.0 is an early pilot release. It has been tested on real laboratory storage, the release is available through PyPI, and the repository carries 380 automated tests that run on Python 3.9 and 3.13 across Linux, macOS and Windows—including a job that runs every module against a bare interpreter, so the zero-dependency guarantee cannot quietly lapse.
What it does not yet have is sufficient testing across independent laboratories, modalities, acquisition systems and naming conventions.
The most useful feedback is a layout it reads incorrectly.
"I gave up at step three" is a genuinely useful result. So is disagreement with the specification for a scientifically sound reason.
Repository: github.com/Elnazkarami/N-DOS
If the underlying problem is familiar to you—whether you want to test, criticize, contribute, collaborate or simply help the project reach laboratories that need it—I would be very glad to hear from you.
All reactions