docs: hidden attributes are platform-only; clarify users cannot declare them#162
Open
dimitri-yatsenko wants to merge 5 commits intomainfrom
Open
docs: hidden attributes are platform-only; clarify users cannot declare them#162dimitri-yatsenko wants to merge 5 commits intomainfrom
dimitri-yatsenko wants to merge 5 commits intomainfrom
Conversation
Hidden attributes (names starting with `_`) were primarily designed for
platform operations — DataJoint itself uses them for `_job_start_time`,
`_job_duration`, `_job_version` on Computed/Imported tables and for the
`_singleton` implementation detail. Some functionality is intentionally
exposed to users (notably: a unique index can reference a hidden column,
making `_params_hash`-style derived columns useful), but the feature is
not intended as a general column-hiding tool.
Reframe section 3.4 around that intent, and replace the previous
behavior table with a verified one drawn from the actual code paths:
- Distinguishes platform-managed (auto-injected) from user-defined.
- Documents the exact filter point (Heading.attributes) and lists every
user-facing surface that consumes it: fetch, proj, joins, dict vs.
string restrictions, insert/update1, repr, describe.
- Calls out that fetch1("_name")/proj("_name") explicitly *is* allowed,
matching the test_hidden_job_metadata.py spec.
- Adds a round-trip caveat for describe(): platform-managed hidden
columns regenerate fine because they're re-injected on declare,
but user-defined hidden columns (like _params_hash) are silently
dropped from describe() output.
- Adds guidance on when to declare a hidden attribute vs. a regular one.
Aligns with #1433 (which made user-defined hidden attributes parsable
in the first place).
Merged
3 tasks
Expand §3.4 with a write caveat covering the three observed behaviors: 1. update1 raises "Attribute '_name' not found" — heading.names is filtered (heading.py:232). 2. insert raises "Field '_name' not in table heading" — Heading.__iter__ walks the filtered view (heading.py:367). 3. insert(..., ignore_extra_fields=True) silently *drops* the hidden key without writing it. Less obvious than the loud error and easy to miss. Also note that platform-managed hidden columns (_job_start_time, etc.) are populated by DataJoint internals via raw SQL during populate() (autopopulate.py:786), not via insert/update1. There is no public-API path to write to a hidden column today; users with a declared hidden column must reach for connection.query() or compute the value inside an auto_populate step. Tracks the write side of the gap that #1441 leaves open.
The previous "when to declare hidden" paragraph allowed too much: backing an index was treated as sufficient reason to hide. It isn't. The clean heuristic is: if application code touches the column (computes it, inserts it, queries on it, wants it in describe() output), it should be a regular attribute. Hidden is for platform/implementation concerns the application code never references — _job_* populated by autopopulate internals, _singleton's implementation pattern, or fields that would actively interfere with natural-join semantics. Use the params_hash-with-unique-index case as a concrete example of when NOT to hide: even though it backs an index, the application code computes and inserts the hash, so it should be regular and let proj() handle visibility at the call site if needed.
Updated to reflect the design decision in datajoint/datajoint-python#1441: the parser keeps rejecting leading-underscore attribute names and now returns a clear DataJointError instead of a cryptic ParseException. Reframe §3.4 around the platform-managed-only intent: - Lead paragraph states up-front that user-defined hidden attributes are not supported, and shows the new error message users will see. - Drop the "User-defined hidden attributes" subsection and the _params_hash hidden example. - Keep the platform-attributes table and the behavior matrix — both are still useful for users encountering platform-managed hidden columns (_job_start_time, etc.) in fetch results, joins, and describe output. - Add an explanation paragraph ("Why users can't declare them") covering the no-write-path / no-round-trip / silent-filter rationale. - Replace the user-defined example with a regular-attribute example (params_hash backing a unique index), demonstrating the recommended pattern: declare as a regular attribute, use proj() at the call site for visibility control.
dimitri-yatsenko
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Reworks
reference/specs/table-declaration.md§3.4 to reflect the design decision in datajoint/datajoint-python#1441: user-defined hidden attributes are not supported, and the parser now returns a clear `DataJointError` (instead of a cryptic `pyparsing.ParseException`) when a definition uses a leading-underscore name.What changed in §3.4
_job_start_time,_job_duration,_job_version,_singleton) is preserved — these are the actual hidden columns users encounter in fetch results, joins, and describe output.describe()round-trip, silent filtering on dict restrictions andinsert(ignore_extra_fields=True)._params_hash-as-hidden example are removed. Replaced with a regular-attribute example showing the recommended pattern: declareparams_hashas a regular column, useproj()at the call site if visibility control is needed.Companion code PR
datajoint/datajoint-python#1441 — replaces the cryptic `pyparsing.ParseException` with the helpful `DataJointError` and adds a unit test asserting that `compile_attribute` rejects leading-underscore names with the new message.
Test plan