Skip to content

DataJoint Library 2.3.6

Latest

Choose a tag to compare

@dimitri-yatsenko dimitri-yatsenko released this 18 Nov 16:54
· 86 commits to master since this release
de9d28a

DataJoint 2.3.6 fixes five security issues; upgrade promptly.

This is a security release on the 2.3 line. It contains 2.3.5, the fixes below, and a higher minimum PyMySQL version (see Dependencies). Every earlier release has at least one of these issues. Releases before 2.0, including 0.14, are no longer supported and will not receive a fix. The bug fixes merged since 2.3.5 will come in 2.3.7.

Upgrade with pip install --upgrade "datajoint>=2.3.6". The conda-forge package follows within about an hour of PyPI.

Security

CVE IDs for these advisories are requested and will be added to each advisory when assigned.

  • Files: GHSA-94rh-cpp4-v56c. Affects 0.12.0 and later. Reported by Artur Chlebicki (@Artur12555).
  • Database TLS: GHSA-pvrw-x8qr-w6v9. Affects 0.12.0 and later. From 0.12.0 through 2.0.x, only MySQL is affected, mainly with PyMySQL 1.1 or earlier.
  • Restrictions: GHSA-pv3r-mxv6-wccj. Affects 0.13.0 and later, and 0.8.0 through 0.10.2.
  • Table definitions and blobs: GHSA-gmj8-m8g8-9wg9. The blob part affects all earlier releases. The alter() part affects 0.12.0 and later.
  • Configuration files: GHSA-88hh-c8vj-9xx9. Affects all earlier releases. Releases before 2.0 are affected in a narrower way.

Behavior changes

Files

  • A <filepath@> value that points outside its store is now refused. This means a path with a .. part, or, for a file store, an absolute path outside the store's location.
    • Inserting such a value raises DataJointError.
    • Rows already stored this way, such as some migrated from 0.14, can still be fetched. Reading or downloading the fetched value raises DataJointError.
    • To accept such paths, set dj.config.allow_filepath_outside_store = True or the environment variable DJ_ALLOW_FILEPATH_OUTSIDE_STORE=true. This turns the check off for every <filepath@> value, so use it only with data you trust.
    • The setting applies only to <filepath@>. The error message says so.
  • Every other path that DataJoint reads from or writes to a store is checked in the same way. This covers the paths for <blob@>, <npy@>, <attach@> and <object@> values. A path with a .. part or a NUL (zero) byte raises DataJointError. On a file store, DataJoint also checks where the full path resolves, so a path must stay inside location even when location is empty or .. There, a path that starts with file:, local: or ~ is refused if it would resolve outside location; otherwise it stays a name inside the store. Paths that DataJoint creates itself always pass.
  • The check compares paths as written and does not follow symbolic links. Suppose a file store's location is a symbolic link, for example /tmp on macOS, which links to /private/tmp. Then an absolute path through the link's target, such as one returned by Path.resolve(), is refused. Give the path relative to the store, or under location exactly as written.
  • An absolute <filepath@> path inside a file store's location is now saved relative to the store. The fetched .path is that relative path, not the string you inserted.
  • The check for the reserved store sections (_hash/ and _schema/ by default) now cleans up the path first and ignores case. Paths such as ./_hash/x or _HASH/x now raise ValueError.
  • A stored attachment name that is empty, . or .., or that contains /, \ or a NUL byte, raises DataJointError on fetch, and nothing is written. Such names, including ones with a backslash, can still be inserted; the error comes on fetch.
    • On Windows, a name containing : is also refused. For example, an attachment named 12:00.txt that was inserted from Linux or macOS cannot be fetched on Windows. Fetch such attachments on Linux or macOS.
  • Fetching an attachment no longer reuses or writes through an existing symbolic link, folder or other special file of the same name in the download folder. The attachment is saved under the next free name (name_1.ext, name_2.ext, ...). A regular file with the same content is still reused.
  • An <object@> path with a .. part is refused, on insert and on fetch.
  • When ObjectRef.download() saves a single file into a folder, it uses only the text after the last / or \ of the stored name. On Linux and macOS, a file stored as we\ird.txt is saved as ird.txt.
  • Downloading an <object@> folder now checks every entry before anything is written. If any entry would land outside destination/<object name>/, the download raises DataJointError and writes no files. Entries whose names contain a backslash, or a : on Windows, are also refused, as are keys with an empty or . segment (such as a//b or a/./b), which earlier releases downloaded with the extra segment collapsed. Downloaded files keep the same layout as before. With fsspec older than 2024.3.1, downloading an empty folder object now raises FileNotFoundError instead of creating an empty folder.

Database TLS

TLS encrypts the connection between DataJoint and the database server.

  • On MySQL, use_tls=True or a use_tls dict now requires TLS. If the server does not offer TLS, the connection fails before the user name and password are sent. In 2.3.5, use_tls=True could connect without encryption.
    • use_tls=True encrypts the connection but does not verify the server's certificate. To verify it, pass use_tls={"ca": "/path/to/ca.pem"}.
    • To connect to a server without TLS, set use_tls=False.
  • On PostgreSQL, DataJoint now uses the ca, cert, key and password keys of a use_tls dict.
    • With ca, DataJoint verifies the server's certificate and host name.
    • If the certificate's name does not match the host, as with many self-signed certificates, add "check_hostname": False. Then only the certificate is verified.
    • Any other key is ignored, and a warning is logged.
    • use_tls=True still requires TLS without verification. Before, a dict did the same.
  • The default, use_tls=None, has changed on both backends.
    • On MySQL, it now uses TLS whenever the server offers it. With PyMySQL 1.1, 2.3.5 did not. It connects without TLS only when the server does not offer TLS.
    • A failed TLS handshake (the step that sets up encryption) is no longer retried without TLS. If TLS cannot be set up with a server, for example one limited to TLS 1.0 or 1.1, set use_tls=False.
    • On PostgreSQL, it tries TLS first. It connects without TLS only if the server does not support TLS, or if its pg_hba.conf does not accept TLS connections from this client. pg_hba.conf is the file that controls which connections the server accepts.
  • A failed login is no longer retried without TLS. The server records one failed attempt instead of two. The warning "SSL connection failed ... Falling back to non-SSL connection" no longer appears.
  • use_tls=False now turns TLS off. Before, the setting never reached the driver. So on PostgreSQL, and on MySQL with PyMySQL 1.2 or later, the connection used TLS whenever the server offered it. Now a server that requires encrypted connections refuses the connection, such as MySQL with require_secure_transport=ON. Remove use_tls=False or set use_tls=True.
  • dj.Instance and dj.Connection now read database.use_tls from the configuration, or the DJ_USE_TLS environment variable, when use_tls is not given. dj.conn() already did this.
    • A setting of false now turns TLS off for them.
    • A setting of true now makes them refuse a server without TLS.
    • Pass use_tls= to override the configuration.
  • dj.conn(use_tls=True) or dj.conn(use_tls={...}) now reconnects an existing connection with that setting, unless the connection already meets it. Before, the argument was ignored.
    • A connection made with a dict meets True. A dict is met only by a connection made with the same dict.
    • Inside a transaction, the reconnect raises DataJointError.
    • dj.conn(use_tls=False) does not change an existing connection. Use reset=True.
    • To keep the existing connection, call dj.conn() without use_tls.

Restrictions

  • A value in a dict, record-array or DataFrame restriction must be one of these:

    • a string, number, boolean or None;
    • a date, datetime or datetime.time;
    • a Decimal or a list;
    • a UUID, for a uuid attribute;
    • a dict, for a JSON path.

    Other values now raise DataJointError. Before, they were written into the query unquoted, and most of them did not give a correct query. They include:

    • bytes and memoryview;
    • numpy.datetime64;
    • datetime.timedelta (PyMySQL returns it for MySQL time columns);
    • 0-d NumPy arrays;
    • tuples and complex numbers;
    • NaN and infinity;
    • arbitrary objects.

    Convert such values first. For example, turn a numpy.datetime64 into a datetime.datetime, and a 0-d array into a number with .item(). If pandas converted a DataFrame's date columns to datetime64, use df.astype(object).

  • A JSON attribute key can end with a return type, as in attr.path:type. That type must now be a standard type: integer, floating-point, decimal, boolean, character, binary, JSON, UUID, date or time. It may have a length or precision, a time-zone qualifier, a character set and a collation.

    • Other types now raise DataJointError in restrictions, proj() and index declarations. Examples are char(10) ascii, money and citext.
    • In a restriction, you can write a condition with another type as an SQL string restriction instead.
  • A key that starts with an attribute name but is not a valid JSON path, such as params.bad path, raises DataJointError. Keys that name no attribute are still ignored.

  • List values, and dict values given for a JSON path, are now written as quoted text and compared with their exact text.

    • On MySQL, a dict for a JSON path whose strings contain a double quote or a backslash can now match rows it did not match before.
    • Before, a list containing strings produced invalid SQL.
  • Restriction values now stay inside their quotes whatever the server's string settings. These settings control how the server reads a backslash in quoted text: sql_mode on MySQL and standard_conforming_strings on PostgreSQL. The quoting change itself does not change which rows match under default settings.

    • On MySQL, a single quote in a value is written as two single quotes (''), not with a backslash. This holds under any sql_mode, including NO_BACKSLASH_ESCAPES. Under NO_BACKSLASH_ESCAPES, a value that contains a backslash, a line break or a NUL byte may not match the rows it should.
    • On PostgreSQL, a list value that contains a backslash is written in PostgreSQL's E'...' form. In that form, a backslash is read the same way whatever the server settings.

Table definitions and blobs

  • Foreign-key references (-> ...) in table definitions are now read as table names and never run as Python code. This applies when a table is declared and in alter().

    • The supported forms are Parent, module.Parent and Master.Part. Each may be in parentheses or followed by ().
    • .proj() is supported with string arguments, such as Parent.proj(new_id='parent_id').
  • Other forms now raise a DataJointError that lists the supported forms:

    • a variable, ... or **dict passed to .proj();
    • a call to any other method or function;
    • a string with an r, u or f prefix;
    • a string split into adjacent parts, such as 'a' 'b';
    • a name that begins with two underscores.

    Rewrite such references in a supported form, for example by writing the names in a projection as string literals.

  • If the error says a name is not defined, make sure the parent table's class is defined or imported in the module where the table is declared. For alter(), you can instead pass the namespace that defines it with context=.

  • In describe() output, line breaks become spaces. This applies to table and column names, comments, default values, types and index columns. In a projected reference, a backslash or double quote in a column name is now escaped.

  • A compressed blob whose data expands beyond the size recorded in its header is now refused before it is fully expanded. This does not limit the memory that a blob can use up to its recorded size.

  • A corrupt compressed blob raises DataJointError instead of zlib.error. Code that catches zlib.error should catch DataJointError.

Configuration files

These changes apply to Linux and macOS.

  • The configuration search now skips some files and folders, and logs a warning for each one.

    • It covers datajoint.json in the working directory and its parent directories.
    • It covers the .secrets directory, /run/secrets/datajoint and the secret files in them.
  • An item is skipped when both of these are true:

    • It is owned by another user (not root).
    • It is in a directory that every user can write, such as /tmp, or below such a directory that lacks the sticky bit. The sticky bit is the t in drwxrwxrwt on /tmp. It stops users from removing or renaming each other's files.

    Files in a shared group directory, including one under a sticky /scratch, are still loaded. When a skipped datajoint.json is found, the search stops there, and files in parent directories are not used.

  • These setups can be affected:

    • shared volumes with mode 0777 that another account fills. One example is a Kubernetes emptyDir written by a helper container that runs as a different user.
    • container images that run under a different user ID, for example on OpenShift. This applies when the files belong to a build user and were made writable by everyone with chmod -R 777.
    • Amazon EFS (a cloud file system) access points whose root directory has mode 0777, when the process runs as a different user;
    • network file system (NFS) shares that map file owners to an anonymous user;
    • the Windows drive in WSL (/mnt/c), when running as root or as another WSL user.
  • To use a skipped file, do one of these:

    • set DJ_CONFIG_FILE to its path;
    • call dj.config.load(path);
    • change its owner with chown;
    • remove write permission for other users from the directory with chmod o-w.

    Supply skipped secrets through environment variables. The skip is logged as a warning, so DJ_LOG_LEVEL=ERROR hides it.

  • A warning is now logged when DataJoint uses a configuration file, secret file or secrets directory that every user can write. Restrict it with chmod o-w.

    • A directory that every user can write but that has the sticky bit produces no warning. An example is a root-owned /run/secrets/datajoint with mode 1777.
    • Each file in such a directory is still checked.
  • The new environment variable DJ_CONFIG_FILE names the configuration file to load at import, with no search.

    • The file is used as given, without the ownership check.
    • The .secrets directory next to that file is also used, under the rules above.
    • A relative path is resolved against the working directory, and ~ is expanded.
    • If it does not name an existing file, import datajoint raises FileNotFoundError.
    • DJ_IGNORE_CONFIG_FILE=true turns it off along with the search.
  • On Windows, the configuration search is unchanged.

Array cache

  • <npy@> arrays from remote stores opened with mmap_mode are now cached in a directory for each user: <tempdir>/datajoint_mmap_<uid>, with mode 0700, so only that user can read it. Here <tempdir> is the system's temporary directory, such as /tmp, and <uid> is your numeric user ID.
    • If that directory exists and other users can access it, DataJoint restricts it to you and empties it.
    • If the directory belongs to another user, DataJoint uses a new private directory for the session.
  • The first such load after upgrading downloads each array again.
  • The old shared directory <tempdir>/datajoint_mmap is no longer used and can be deleted.
  • On Windows, the cache stays in <tempdir>/datajoint_mmap, which is inside each user's own temporary folder.

Dependencies

  • pymysql >= 1.1.1, up from >= 0.7.2, because of CVE-2024-36039 in PyMySQL.
  • Nothing else changes.