Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merge table multi-source error specificity #609

Closed
MichaelCoulter opened this issue Jul 28, 2023 · 3 comments · Fixed by #617
Closed

Merge table multi-source error specificity #609

MichaelCoulter opened this issue Jul 28, 2023 · 3 comments · Fixed by #617
Assignees

Comments

@MichaelCoulter
Copy link
Collaborator

MichaelCoulter commented Jul 28, 2023

i'm trying to get position (to get speed) for ripple detection.

this will find a single entry in IntervalPositionInfo
IntervalPositionInfo() & {"nwb_file_name": 'tonks20211027_.nwb'} & {'interval_list_name':'pos 1 valid times'}

but when i try this:

nwb_file_name = 'tonks20211027_.nwb'
interval_list_name = "pos 1 valid times"

pos_key = PositionOutput.merge_get_part({
        "nwb_file_name": nwb_file_name,
        "position_info_param_name": "default_decoding",
        "interval_list_name": interval_list_name,
    }).fetch1("KEY")
(PositionOutput & pos_key).fetch1_dataframe()`

i get this error:

ValueError                                Traceback (most recent call last)
Cell In [18], line 5
      2 nwb_file_name = 'tonks20211027_.nwb'
      3 interval_list_name = "pos 1 valid times"
----> 5 pos_key = PositionOutput.merge_get_part({
      6         "nwb_file_name": nwb_file_name,
      7         "position_info_param_name": "default_decoding",
      8         "interval_list_name": interval_list_name,
      9     }).fetch1("KEY")
     10 (PositionOutput & pos_key).fetch1_dataframe()

File ~/spyglass/src/spyglass/utils/dj_merge_tables.py:543, in Merge.merge_get_part(cls, restriction, join_master, restrict_part, multi_source)
    532 sources = [
    533     to_camel_case(n.split("__")[-1].strip("`"))  # friendly part name
    534     for n in cls._merge_restrict_parts(
   (...)
    539     )
    540 ]
    542 if not multi_source and len(sources) != 1:
--> 543     raise ValueError(
    544         f"Found multiple potential parts: {sources}\n\t"
    545         + "Try adding a restriction before invoking `get_part`.\n\t"
    546         + "Or permitting multiple sources with `multi_source=True`."
    547     )
    549 parts = [
    550     getattr(cls, source)().restrict(restriction)
    551     if restrict_part  # Re-apply restriction or don't
    552     else getattr(cls, source)()
    553     for source in sources
    554 ]
    555 if join_master:

ValueError: Found multiple potential parts: []
	Try adding a restriction before invoking `get_part`.
	Or permitting multiple sources with `multi_source=True`.

any ideas how to have a more specific query? thanks.

@dpeg22
Copy link
Collaborator

dpeg22 commented Jul 28, 2023

Hi Mike, it looks like that entry doesn't exist in PositionOutput.CommonPos, which is the PositionOutput part table for IntervalPositionInfo, so that is something you'll need to do prior to querying PositionOutput.
@CBroz1 might have better suggestions, but if you know your entries are coming from IntervalPositionInfo and they've been inserted into PositionOutput, then you could run:

pos_key = (PositionOutput.CommonPos & {
    "nwb_file_name": nwb_file_name,
    "position_info_param_name": "default_decoding", 
    "interval_list_name": interval_list_name,
}).fetch1("KEY")

@MichaelCoulter
Copy link
Collaborator Author

Here is a command to insert entries from IntervalPositionInfo into PositionOutput

position_insert = (IntervalPositionInfo & {'nwb_file_name':'tonks20211027_.nwb'}).fetch(as_dict=True) PositionOutput._merge_insert(position_insert, part_name="CommonPos")

@CBroz1 CBroz1 changed the title trying to fetch position, get error about multiple parts Merge table multi-source error specificity Jul 28, 2023
@CBroz1
Copy link
Member

CBroz1 commented Jul 28, 2023

Reopening to track adding error specificity to dj_merge_tables.py:544, to

f"Found {len(sources)} potential part(s): {sources}\n\t" 

@CBroz1 CBroz1 reopened this Jul 28, 2023
@CBroz1 CBroz1 self-assigned this Jul 28, 2023
CBroz1 added a commit to CBroz1/spyglass that referenced this issue Aug 3, 2023
CBroz1 added a commit to CBroz1/spyglass that referenced this issue Aug 4, 2023
@CBroz1 CBroz1 linked a pull request Aug 4, 2023 that will close this issue
@edeno edeno closed this as completed in #617 Aug 4, 2023
edeno pushed a commit that referenced this issue Aug 4, 2023
* Restructure for mkdocs

* Minor docstring edits for mkdocs

* Adjust installer docs

* Permit json config

* Update docstrings for mkdocs. Add images

* Minor fixes. Rename publish action. Add changelog notes

* hard wrap changes, markdownlint

* blackify

* Fix version cmd. Minor doc wording

* mkdocstrings require empty inits to identify submodules

* 🤦‍♂️ Publish docs CI/CD `main`->`master`

* Edit `get_part`; Add `merge_fetch`

* Spelling fixes

* Refactor restrict_parts. Adjust mkdocs nav.

* Docs adjust for Merge tables

* Fix `merge_get_part`

* Use hatch for docs version

* Update changelog

* Typo

* Spellcheck config

* WIP: dict/str ristrict consistency

* Normalize restriction/classmeth. Add notes on why

* Typos

* Docs publish on tag

* Edit changelog: Add links, patch version bump

* 🧪 Test gh-actions debug

* get-part multi-source flag

* Add mutual exclusivity flag from pos branch

* See details. Notebook work, config overhaul

gitignore: add exclude example config
mkdocs: new notebook names
notebooks: complete revamp for minirec data and more links to docs
init: add new load_config, isort imports
common_lab:
	- adjust to accept names in Last, First format (nwb-compliant)
	- continue to use First Last name structure in database - yes?
common_nwb: use new load_config, change `assert` to `raise`
insert_sessions: permit paths, use file name, use raw dir
storage_dirs: remove redundant funcs for base_dir
settings: implement new base_dir system
	- allows base/raw/etc to be independent
	- defaults to dj.config, then env vars, then sets default rel paths

* WIP: fix failing tests related to base_dir edits

* underscore-prefix Merge. Linter fixes

* See Details. Notebook overhaul

- dj_config: accept base dir as arg, refactor for single responsibility
- mkdocs, installation.md
  - condense installation information to single page
  - reference new notebook
  - remove local and production subpages due to redundancy
- environment and env_position.yml: add install current dir to avoid
  additional step in installation process
- notebooks: rewrite with Docker optional and minirec as demo data
- common_lab: raise error for invalid name
- common_position: get raw dir from settings, not hardcode
- settings.py: Should this be a class with properties?
  - add options for kachery dirs set via same dj_config mechanism
  - add raw_dir helper function

* remove note to self

* WIP notebook edits

* Revise 04_LFP nb

* Fix #614

* Fix #609
edeno pushed a commit that referenced this issue Oct 10, 2023
* Restructure for mkdocs

* Minor docstring edits for mkdocs

* Adjust installer docs

* Permit json config

* Update docstrings for mkdocs. Add images

* Minor fixes. Rename publish action. Add changelog notes

* hard wrap changes, markdownlint

* blackify

* Fix version cmd. Minor doc wording

* mkdocstrings require empty inits to identify submodules

* 🤦‍♂️ Publish docs CI/CD `main`->`master`

* Edit `get_part`; Add `merge_fetch`

* Spelling fixes

* Refactor restrict_parts. Adjust mkdocs nav.

* Docs adjust for Merge tables

* Fix `merge_get_part`

* Use hatch for docs version

* Update changelog

* Typo

* Spellcheck config

* WIP: dict/str ristrict consistency

* Normalize restriction/classmeth. Add notes on why

* Typos

* Docs publish on tag

* Edit changelog: Add links, patch version bump

* 🧪 Test gh-actions debug

* get-part multi-source flag

* Add mutual exclusivity flag from pos branch

* See details. Notebook work, config overhaul

gitignore: add exclude example config
mkdocs: new notebook names
notebooks: complete revamp for minirec data and more links to docs
init: add new load_config, isort imports
common_lab:
	- adjust to accept names in Last, First format (nwb-compliant)
	- continue to use First Last name structure in database - yes?
common_nwb: use new load_config, change `assert` to `raise`
insert_sessions: permit paths, use file name, use raw dir
storage_dirs: remove redundant funcs for base_dir
settings: implement new base_dir system
	- allows base/raw/etc to be independent
	- defaults to dj.config, then env vars, then sets default rel paths

* WIP: fix failing tests related to base_dir edits

* underscore-prefix Merge. Linter fixes

* See Details. Notebook overhaul

- dj_config: accept base dir as arg, refactor for single responsibility
- mkdocs, installation.md
  - condense installation information to single page
  - reference new notebook
  - remove local and production subpages due to redundancy
- environment and env_position.yml: add install current dir to avoid
  additional step in installation process
- notebooks: rewrite with Docker optional and minirec as demo data
- common_lab: raise error for invalid name
- common_position: get raw dir from settings, not hardcode
- settings.py: Should this be a class with properties?
  - add options for kachery dirs set via same dj_config mechanism
  - add raw_dir helper function

* remove note to self

* WIP notebook edits

* Revise 04_LFP nb

* Reorder/revise notebooks; #609

* Notebook formatting

* Remove old

* jupytext backup note

* Blackify py scrips. Continue config changes

* WIP: notebooks, plus improved merge_delete_downstream

* WIP: PositionSource add part table

* Refactor trodes position #613

* WIP: Fix Trodes Video

* WIP: Spellcheck. Remove debug params. Remove assigned lambda E713

* WIP: Pass tests. Remove codespell offending link

* WIP: blackify

* Selective fetch from cbroz/master

* Fetch additional file from cbroz1/master to pass CI/CD

* Add RawPos fetch method implementations. Object -> PosObject

* Refactor PosIntervalMap helpers

* Revert typo

* Bugfixes for ripple

* Blackify

* WIP: minor edits

* Add restriction to fetch1_dataframe

* Update Trodes notebook, revise others

* Set pos id default for migration. Rename Trodes params

* Fix typos

* Spelling; Jupytext sync; Blackify

* Edit gitignore for new notebook numbering

* Update changelog and notebooks

* Refactor position helpers

* Minor position_trodes fixes
edeno pushed a commit that referenced this issue Oct 12, 2023
* Restructure for mkdocs

* Minor docstring edits for mkdocs

* Adjust installer docs

* Permit json config

* Update docstrings for mkdocs. Add images

* Minor fixes. Rename publish action. Add changelog notes

* hard wrap changes, markdownlint

* blackify

* Fix version cmd. Minor doc wording

* mkdocstrings require empty inits to identify submodules

* 🤦‍♂️ Publish docs CI/CD `main`->`master`

* Edit `get_part`; Add `merge_fetch`

* Spelling fixes

* Refactor restrict_parts. Adjust mkdocs nav.

* Docs adjust for Merge tables

* Fix `merge_get_part`

* Use hatch for docs version

* Update changelog

* Typo

* Spellcheck config

* WIP: dict/str ristrict consistency

* Normalize restriction/classmeth. Add notes on why

* Typos

* Docs publish on tag

* Edit changelog: Add links, patch version bump

* 🧪 Test gh-actions debug

* get-part multi-source flag

* Add mutual exclusivity flag from pos branch

* See details. Notebook work, config overhaul

gitignore: add exclude example config
mkdocs: new notebook names
notebooks: complete revamp for minirec data and more links to docs
init: add new load_config, isort imports
common_lab:
	- adjust to accept names in Last, First format (nwb-compliant)
	- continue to use First Last name structure in database - yes?
common_nwb: use new load_config, change `assert` to `raise`
insert_sessions: permit paths, use file name, use raw dir
storage_dirs: remove redundant funcs for base_dir
settings: implement new base_dir system
	- allows base/raw/etc to be independent
	- defaults to dj.config, then env vars, then sets default rel paths

* WIP: fix failing tests related to base_dir edits

* underscore-prefix Merge. Linter fixes

* See Details. Notebook overhaul

- dj_config: accept base dir as arg, refactor for single responsibility
- mkdocs, installation.md
  - condense installation information to single page
  - reference new notebook
  - remove local and production subpages due to redundancy
- environment and env_position.yml: add install current dir to avoid
  additional step in installation process
- notebooks: rewrite with Docker optional and minirec as demo data
- common_lab: raise error for invalid name
- common_position: get raw dir from settings, not hardcode
- settings.py: Should this be a class with properties?
  - add options for kachery dirs set via same dj_config mechanism
  - add raw_dir helper function

* remove note to self

* WIP notebook edits

* Revise 04_LFP nb

* Reorder/revise notebooks; #609

* Notebook formatting

* Remove old

* jupytext backup note

* Blackify py scrips. Continue config changes

* Refactor common

* WIP: notebooks, plus improved merge_delete_downstream

* WIP: PositionSource add part table

* Refactor trodes position #613

* WIP: Fix Trodes Video

* WIP: Spellcheck. Remove debug params. Remove assigned lambda E713

* WIP: Pass tests. Remove codespell offending link

* WIP: blackify

* Selective fetch from cbroz/master

* Fetch additional file from cbroz1/master to pass CI/CD

* Add RawPos fetch method implementations. Object -> PosObject

* Refactor PosIntervalMap helpers

* Revert typo

* Bugfixes for ripple

* Blackify

* WIP: minor edits

* Add restriction to fetch1_dataframe

* Update Trodes notebook, revise others

* Set pos id default for migration. Rename Trodes params

* Fix typos

* Spelling; Jupytext sync; Blackify

* Edit gitignore for new notebook numbering

* Update changelog and notebooks

* Refactor position helpers

* Minor position_trodes fixes

* Docs fixes

* Minor docs nav rename

* Fix typos

* Config to OOP. And #365

* Fix failing test from prev commit. Also #585

* Fixes from review comments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants