Repository navigation
DataJoint 2.3.6 fixes five security issues; upgrade promptly.
This is a security release on the 2.3 line. It contains 2.3.5, the fixes below, and a higher minimum PyMySQL version (see Dependencies). Every earlier release has at least one of these issues. Releases before 2.0, including 0.14, are no longer supported and will not receive a fix. The bug fixes merged since 2.3.5 will come in 2.3.7.
Upgrade with pip install --upgrade "datajoint>=2.3.6". The conda-forge package follows within about an hour of PyPI.
Security
CVE IDs for these advisories are requested and will be added to each advisory when assigned.
- Files: GHSA-94rh-cpp4-v56c. Affects 0.12.0 and later. Reported by Artur Chlebicki (@Artur12555).
- Database TLS: GHSA-pvrw-x8qr-w6v9. Affects 0.12.0 and later. From 0.12.0 through 2.0.x, only MySQL is affected, mainly with PyMySQL 1.1 or earlier.
- Restrictions: GHSA-pv3r-mxv6-wccj. Affects 0.13.0 and later, and 0.8.0 through 0.10.2.
- Table definitions and blobs: GHSA-gmj8-m8g8-9wg9. The blob part affects all earlier releases. The
alter()part affects 0.12.0 and later. - Configuration files: GHSA-88hh-c8vj-9xx9. Affects all earlier releases. Releases before 2.0 are affected in a narrower way.
Behavior changes
Files
- A
<filepath@>value that points outside its store is now refused. This means a path with a..part, or, for afilestore, an absolute path outside the store'slocation.- Inserting such a value raises
DataJointError. - Rows already stored this way, such as some migrated from 0.14, can still be fetched. Reading or downloading the fetched value raises
DataJointError. - To accept such paths, set
dj.config.allow_filepath_outside_store = Trueor the environment variableDJ_ALLOW_FILEPATH_OUTSIDE_STORE=true. This turns the check off for every<filepath@>value, so use it only with data you trust. - The setting applies only to
<filepath@>. The error message says so.
- Inserting such a value raises
- Every other path that DataJoint reads from or writes to a store is checked in the same way. This covers the paths for
<blob@>,<npy@>,<attach@>and<object@>values. A path with a..part or a NUL (zero) byte raisesDataJointError. On afilestore, DataJoint also checks where the full path resolves, so a path must stay insidelocationeven whenlocationis empty or.. There, a path that starts withfile:,local:or~is refused if it would resolve outsidelocation; otherwise it stays a name inside the store. Paths that DataJoint creates itself always pass. - The check compares paths as written and does not follow symbolic links. Suppose a file store's
locationis a symbolic link, for example/tmpon macOS, which links to/private/tmp. Then an absolute path through the link's target, such as one returned byPath.resolve(), is refused. Give the path relative to the store, or underlocationexactly as written. - An absolute
<filepath@>path inside a file store'slocationis now saved relative to the store. The fetched.pathis that relative path, not the string you inserted. - The check for the reserved store sections (
_hash/and_schema/by default) now cleans up the path first and ignores case. Paths such as./_hash/xor_HASH/xnow raiseValueError. - A stored attachment name that is empty,
.or.., or that contains/,\or a NUL byte, raisesDataJointErroron fetch, and nothing is written. Such names, including ones with a backslash, can still be inserted; the error comes on fetch.- On Windows, a name containing
:is also refused. For example, an attachment named12:00.txtthat was inserted from Linux or macOS cannot be fetched on Windows. Fetch such attachments on Linux or macOS.
- On Windows, a name containing
- Fetching an attachment no longer reuses or writes through an existing symbolic link, folder or other special file of the same name in the download folder. The attachment is saved under the next free name (
name_1.ext,name_2.ext, ...). A regular file with the same content is still reused. - An
<object@>path with a..part is refused, on insert and on fetch. - When
ObjectRef.download()saves a single file into a folder, it uses only the text after the last/or\of the stored name. On Linux and macOS, a file stored aswe\ird.txtis saved asird.txt. - Downloading an
<object@>folder now checks every entry before anything is written. If any entry would land outsidedestination/<object name>/, the download raisesDataJointErrorand writes no files. Entries whose names contain a backslash, or a:on Windows, are also refused, as are keys with an empty or.segment (such asa//bora/./b), which earlier releases downloaded with the extra segment collapsed. Downloaded files keep the same layout as before. With fsspec older than 2024.3.1, downloading an empty folder object now raisesFileNotFoundErrorinstead of creating an empty folder.
Database TLS
TLS encrypts the connection between DataJoint and the database server.
- On MySQL,
use_tls=Trueor ause_tlsdict now requires TLS. If the server does not offer TLS, the connection fails before the user name and password are sent. In 2.3.5,use_tls=Truecould connect without encryption.use_tls=Trueencrypts the connection but does not verify the server's certificate. To verify it, passuse_tls={"ca": "/path/to/ca.pem"}.- To connect to a server without TLS, set
use_tls=False.
- On PostgreSQL, DataJoint now uses the
ca,cert,keyandpasswordkeys of ause_tlsdict.- With
ca, DataJoint verifies the server's certificate and host name. - If the certificate's name does not match the host, as with many self-signed certificates, add
"check_hostname": False. Then only the certificate is verified. - Any other key is ignored, and a warning is logged.
use_tls=Truestill requires TLS without verification. Before, a dict did the same.
- With
- The default,
use_tls=None, has changed on both backends.- On MySQL, it now uses TLS whenever the server offers it. With PyMySQL 1.1, 2.3.5 did not. It connects without TLS only when the server does not offer TLS.
- A failed TLS handshake (the step that sets up encryption) is no longer retried without TLS. If TLS cannot be set up with a server, for example one limited to TLS 1.0 or 1.1, set
use_tls=False. - On PostgreSQL, it tries TLS first. It connects without TLS only if the server does not support TLS, or if its
pg_hba.confdoes not accept TLS connections from this client.pg_hba.confis the file that controls which connections the server accepts.
- A failed login is no longer retried without TLS. The server records one failed attempt instead of two. The warning "SSL connection failed ... Falling back to non-SSL connection" no longer appears.
use_tls=Falsenow turns TLS off. Before, the setting never reached the driver. So on PostgreSQL, and on MySQL with PyMySQL 1.2 or later, the connection used TLS whenever the server offered it. Now a server that requires encrypted connections refuses the connection, such as MySQL withrequire_secure_transport=ON. Removeuse_tls=Falseor setuse_tls=True.dj.Instanceanddj.Connectionnow readdatabase.use_tlsfrom the configuration, or theDJ_USE_TLSenvironment variable, whenuse_tlsis not given.dj.conn()already did this.- A setting of
falsenow turns TLS off for them. - A setting of
truenow makes them refuse a server without TLS. - Pass
use_tls=to override the configuration.
- A setting of
dj.conn(use_tls=True)ordj.conn(use_tls={...})now reconnects an existing connection with that setting, unless the connection already meets it. Before, the argument was ignored.- A connection made with a dict meets
True. A dict is met only by a connection made with the same dict. - Inside a transaction, the reconnect raises
DataJointError. dj.conn(use_tls=False)does not change an existing connection. Usereset=True.- To keep the existing connection, call
dj.conn()withoutuse_tls.
- A connection made with a dict meets
Restrictions
-
A value in a dict, record-array or DataFrame restriction must be one of these:
- a string, number, boolean or
None; - a date,
datetimeordatetime.time; - a
Decimalor a list; - a UUID, for a
uuidattribute; - a dict, for a JSON path.
Other values now raise
DataJointError. Before, they were written into the query unquoted, and most of them did not give a correct query. They include:- bytes and
memoryview; numpy.datetime64;datetime.timedelta(PyMySQL returns it for MySQLtimecolumns);- 0-d NumPy arrays;
- tuples and complex numbers;
- NaN and infinity;
- arbitrary objects.
Convert such values first. For example, turn a
numpy.datetime64into adatetime.datetime, and a 0-d array into a number with.item(). If pandas converted a DataFrame's date columns todatetime64, usedf.astype(object). - a string, number, boolean or
-
A JSON attribute key can end with a return type, as in
attr.path:type. That type must now be a standard type: integer, floating-point, decimal, boolean, character, binary, JSON, UUID, date or time. It may have a length or precision, a time-zone qualifier, a character set and a collation.- Other types now raise
DataJointErrorin restrictions,proj()and index declarations. Examples arechar(10) ascii,moneyandcitext. - In a restriction, you can write a condition with another type as an SQL string restriction instead.
- Other types now raise
-
A key that starts with an attribute name but is not a valid JSON path, such as
params.bad path, raisesDataJointError. Keys that name no attribute are still ignored. -
List values, and dict values given for a JSON path, are now written as quoted text and compared with their exact text.
- On MySQL, a dict for a JSON path whose strings contain a double quote or a backslash can now match rows it did not match before.
- Before, a list containing strings produced invalid SQL.
-
Restriction values now stay inside their quotes whatever the server's string settings. These settings control how the server reads a backslash in quoted text:
sql_modeon MySQL andstandard_conforming_stringson PostgreSQL. The quoting change itself does not change which rows match under default settings.- On MySQL, a single quote in a value is written as two single quotes (
''), not with a backslash. This holds under anysql_mode, includingNO_BACKSLASH_ESCAPES. UnderNO_BACKSLASH_ESCAPES, a value that contains a backslash, a line break or a NUL byte may not match the rows it should. - On PostgreSQL, a list value that contains a backslash is written in PostgreSQL's
E'...'form. In that form, a backslash is read the same way whatever the server settings.
- On MySQL, a single quote in a value is written as two single quotes (
Table definitions and blobs
-
Foreign-key references (
-> ...) in table definitions are now read as table names and never run as Python code. This applies when a table is declared and inalter().- The supported forms are
Parent,module.ParentandMaster.Part. Each may be in parentheses or followed by(). .proj()is supported with string arguments, such asParent.proj(new_id='parent_id').
- The supported forms are
-
Other forms now raise a
DataJointErrorthat lists the supported forms:- a variable,
...or**dictpassed to.proj(); - a call to any other method or function;
- a string with an
r,uorfprefix; - a string split into adjacent parts, such as
'a' 'b'; - a name that begins with two underscores.
Rewrite such references in a supported form, for example by writing the names in a projection as string literals.
- a variable,
-
If the error says a name is not defined, make sure the parent table's class is defined or imported in the module where the table is declared. For
alter(), you can instead pass the namespace that defines it withcontext=. -
In
describe()output, line breaks become spaces. This applies to table and column names, comments, default values, types and index columns. In a projected reference, a backslash or double quote in a column name is now escaped. -
A compressed blob whose data expands beyond the size recorded in its header is now refused before it is fully expanded. This does not limit the memory that a blob can use up to its recorded size.
-
A corrupt compressed blob raises
DataJointErrorinstead ofzlib.error. Code that catcheszlib.errorshould catchDataJointError.
Configuration files
These changes apply to Linux and macOS.
-
The configuration search now skips some files and folders, and logs a warning for each one.
- It covers
datajoint.jsonin the working directory and its parent directories. - It covers the
.secretsdirectory,/run/secrets/datajointand the secret files in them.
- It covers
-
An item is skipped when both of these are true:
- It is owned by another user (not root).
- It is in a directory that every user can write, such as
/tmp, or below such a directory that lacks the sticky bit. The sticky bit is thetindrwxrwxrwton/tmp. It stops users from removing or renaming each other's files.
Files in a shared group directory, including one under a sticky
/scratch, are still loaded. When a skippeddatajoint.jsonis found, the search stops there, and files in parent directories are not used. -
These setups can be affected:
- shared volumes with mode 0777 that another account fills. One example is a Kubernetes
emptyDirwritten by a helper container that runs as a different user. - container images that run under a different user ID, for example on OpenShift. This applies when the files belong to a build user and were made writable by everyone with
chmod -R 777. - Amazon EFS (a cloud file system) access points whose root directory has mode 0777, when the process runs as a different user;
- network file system (NFS) shares that map file owners to an anonymous user;
- the Windows drive in WSL (
/mnt/c), when running as root or as another WSL user.
- shared volumes with mode 0777 that another account fills. One example is a Kubernetes
-
To use a skipped file, do one of these:
- set
DJ_CONFIG_FILEto its path; - call
dj.config.load(path); - change its owner with
chown; - remove write permission for other users from the directory with
chmod o-w.
Supply skipped secrets through environment variables. The skip is logged as a warning, so
DJ_LOG_LEVEL=ERRORhides it. - set
-
A warning is now logged when DataJoint uses a configuration file, secret file or secrets directory that every user can write. Restrict it with
chmod o-w.- A directory that every user can write but that has the sticky bit produces no warning. An example is a root-owned
/run/secrets/datajointwith mode 1777. - Each file in such a directory is still checked.
- A directory that every user can write but that has the sticky bit produces no warning. An example is a root-owned
-
The new environment variable
DJ_CONFIG_FILEnames the configuration file to load at import, with no search.- The file is used as given, without the ownership check.
- The
.secretsdirectory next to that file is also used, under the rules above. - A relative path is resolved against the working directory, and
~is expanded. - If it does not name an existing file,
import datajointraisesFileNotFoundError. DJ_IGNORE_CONFIG_FILE=trueturns it off along with the search.
-
On Windows, the configuration search is unchanged.
Array cache
<npy@>arrays from remote stores opened withmmap_modeare now cached in a directory for each user:<tempdir>/datajoint_mmap_<uid>, with mode 0700, so only that user can read it. Here<tempdir>is the system's temporary directory, such as/tmp, and<uid>is your numeric user ID.- If that directory exists and other users can access it, DataJoint restricts it to you and empties it.
- If the directory belongs to another user, DataJoint uses a new private directory for the session.
- The first such load after upgrading downloads each array again.
- The old shared directory
<tempdir>/datajoint_mmapis no longer used and can be deleted. - On Windows, the cache stays in
<tempdir>/datajoint_mmap, which is inside each user's own temporary folder.
Dependencies
pymysql >= 1.1.1, up from>= 0.7.2, because of CVE-2024-36039 in PyMySQL.- Nothing else changes.