sync master #2
Open
sync master #2
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
There were a couple of cases where pseudo built against GLIBC_2.7 or newer was ending up with dependencies on symbols which required GLIBC_2.7. With these gone, it appears that a libpseudo.so can be used on an older host in some cases. None were particularly important or intentional: 1. pseudo_util was conditionally calling open() with only two arguments, which can invoke a new __open2() function in some systems. Don't care, and the docs specifically state that the mode argument is "ignored" when O_CREAT is absent, so it's not necessary to omit it. 2. The calls to sscanf/fscanf in pseudo_client.c were getting translated into a special new iso_c99 sscanf/fscanf, and we don't care because we're not using those features; #define _GNU_SOURCE suppresses the extra-compliant behavior. Signed-off-by: seebs <peter.seebach@windriver.com>
There's a few circumstances where sqlite's libdir isn't the same as the directory name we want to use for libpseudo.so, so make it a separate setting. This may obviate the need for --with-static-sqlite. Signed-off-by: seebs <peter.seebach@windriver.com>
Fixed up the libdir stuff in a saner way, fixed the @GLIBC_2.7 thing, want to avoid any risk of someone having a stale archive.
The automatic path fixups invoked for names which end in the string "path" was still applying to link(), which then called linkat(), which would do the same path fixups; if you were chrooted, this would produce bogus paths. On systems which actually have linkat(), this would produce the even more mysterious behavior that the link would succeed, but the following stat would fail. Solution: Change the wrapfuncs prototypes for link() so it doesn't invoke automatic path name fixups.
wrap_linkat() was trying to avoid "redundantly" expanding paths before calling real_linkat(). Which is fine when you're not using an absolute path in a chroot environment, but if you are, it ends up calling the real syscall with the absolute path and no chroot prefix. General observation: All the *at() implementations are expanding paths into absolute paths, then dutifully calling real_*at() functions with them anyway. This is silly. Added a note to Futures.txt to fix it some day. In the mean time, linkat() is fixed correctly; it always expands paths, does so exactly once, and then uses the underlying link() call because it doesn't need special processing of directory fds anymore. Also fixed errno stashing to reduce the risk that link() will change errno in a circumstance where it doesn't actually fail.
It turns out that file databases don't get very large, and that sqlite3 can be quite fast with an in-memory database. It also turns out that dumping the database to disk on exit (or during idle times) is pretty cheap compared to constant updates. So: We add "--enable-memory-db", which defaults to on if you have sqlite 3.7 or later, and off for 3.6 (because 3.6 has horrible performance with in-memory db on some hosts we tried).
The openembedded build, at least with RPM or SMART, is heavily affected by the cost of calling fsync or fdatasync on package databases all the time. Gosh, wouldn't it be nice if we could suppress that without making dozens of highly intrusive and risky changes into RPM, various database packages, and so on? Yes, yes it would. If only there were a program which could intercept system calls and change their behavior! Enter --enable-force-async. There are now wrappers for fsync, fdatasync, and a few related functions. If --enable-force-async is set, these wrappers instantly return 0, even if PSEUDO_DISABLED is set. And with any luck, bitbake will now perform a bit better. Credit for this insight goes to Richard Purdie. I've reimplemented this to add the configure option, and make the fsync suppression work even when PSEUDO_DISABLED is set.
Most pseudo operations don't actually USE the server's response. So why wait for a response? This patch introduces a new message type, PSEUDO_MSG_FASTOP. It also tags pseudo operation types with whether or not they need to give a response. This requires updates to maketables to allow non-string types for additional columns, and the addition of some quotes to the SQL query enums/query_type.in table. A few routines are altered to change their behavior and whether or not they perform a stat operation. The only operations that do wait are OP_FSTAT and OP_STAT, OP_MKNOD, and OP_MAY_UNLINK. Rationale: You can't query the server for replacement information and not wait for it. Makes no sense. There's extra checking in mknod, because we really do want to fail out if we couldn't do that -- that implies that we haven't created a thing that will look like a node. The result from OP_MAY_UNLINK is checked because it's used to determine whether we need to send a DID_UNLINK or CANCEL_UNLINK. It might be cheaper to send two messages without waiting than to send one, wait, and maybe send another, but I don't want to send invalid messages. This is highly experimental.
Darwin's off_t is a 64-bit type, so there's no off64_t. Also, there's an uninitialized variable usage in unlinkat which LLVM catches. Signed-off-by: Peter Seebach <peter.seebach@windriver.com>
Some filesystems have buggy semantics where stat(2) will return incorrect sizes for files for a while after some changes, sometimes, unless they've been fsync'd. We still want to disable fsync most of the time, but enabling it for specific programs can be useful. Signed-off-by: Peter Seebach <peter.seebach@windriver.com>
No idea how this survived so long, but the clone() syscall prototype on RHEL 4.7 doesn't have the "..." for additional arguments, so we can't pass them. Also had unused variables that would otherwise have been being filled in from va_args and passed. But there's no extra args to pass. Interestingly, this contradicts the clone() man page in RHEL 4.7. If you have problems with clone() there, that's probably why.
This is a moderately intrusive change. The basic overall effect: Debugging messages are now controlled, not by a numeric "level", but by a series of flags, which are expressed as a string of letters. Each flag has a single-letter form used for string specifications, a name, a description, a numeric value (1 through N), and a flag value (which is 1 << the numeric value). (This does mean that no flag has the value 1, so we only have 31 bits available. Tiny violins play.) The other significant change is that the pseudo_debug calls are now implemented with a do/while macro containing a conditional, so that computationally-expensive arguments are never evaluated if the corresponding debug flags weren't set. The assumption is that in the vast majority of cases (specifically, all of them so far) the debug flags for a given call are a compile-time constant, so the nested conditional will never actually show up in code when compiled with optimization; we'll just see the appropriate conditional test. The VERBOSE flag is magical, in that if the VERBOSE flag is used in a message, the debug flags have to have both VERBOSE and at least one other flag for the call to be made. This should dramatically improve performance for a lot of cases without as much need for PSEUDO_NDEBUG, and improve the ability of users to get coherent debugging output that means something and is relevant to a given case. It's also intended to set the stage for future development work involving improving the clarity and legibility of pseudo's diagnostic messages in general. Old things which used numeric values for PSEUDO_DEBUG will sort of continue to work, though they will almost always be less verbose than they used to. There should probably be a pass through adding "| PDBGF_CONSISTENCY" to a lot of the messages that are specific to some other type.
In some cases, we'd rather pseudo fail than fall back to using /etc/passwd or /etc/group. Make the determination of what to fall back to when neither PSEUDO_PASSWD nor a chroot directory contains passwd/group files controllable by a configure-time flag, controlled by --with-passwd-fallback= or --without-passwd-fallback.
Ports can provide pseudo_wrappers.c or portdefs.h, and individual functions have implementations. These dependencies aren't known until post-configure. Make the Makefile include two sub-Makefiles which can be updated by makewrappers. Signed-off-by: Peter Seebach <peter.seebach@windriver.com>
When invoked as a non-daemon, either with a command or to get a shell, shut the server down automatically. This is currently implemented by sending a shutdown request; I also considered adding an environment flag for "shut down when there's no clients", but this was simpler. Main reason this is useful: In development, it's often the case that I want to run a single command under pseudo, then check the database directly with sqlite. Signed-off-by: Peter Seebach <peter.seebach@windriver.com>
Initial, incomplete, support for extended attributes. Extended attributes are implemented fairly naively, using a second table in the file database using the primary file table's id as a foreign key. The ON DELETE CASCADE behavior requires sqlite 3.6.19 or later with foreign key and trigger support compiled in. To reduce round-trips, the client does not check for existing attributes, but rather, sends three distinct set messages; OP_SET_XATTR, OP_CREATE_XATTR, OP_REPLACE_XATTR. A SET message always succeeds, a CREATE fails if the attribute already exists, and a REPLACE fails if the attribute does not already exist. The /* flags */ feature of makewrappers is used to correct path names appropriately, so all functions are already working with complete paths, and can always use functions that work on links; if they were supposed to dereference, the path fixup code got that. The xattr support is enabled, for now, conditional on whether getfattr --help succeeds. Not yet implemented: Translation for system.posix_acl_access, which is used by "cp -a" (or "cp --preserve-all") on some systems to try to copy modes. Signed-off-by: Peter Seebach <peter.seebach@windriver.com>
Issue wrpseudo#1: If an operation came in for an item with no path provided by the wrapper, the client would not construct the combined "path" value. Fixed, and missing paths are now consistently handled as 0-byte paths. Issue wrpseudo#2: The database code was assuming the values were strings, and ignoring a specified length. Issue wrpseudo#3: The computation of the length of the stored value was off by one, because it was including the extra terminating null the client added in case the value was a path. With this in place, "cp -a" on CentOS is consistently duplicating the system.posix_acl_access fields as expected, but unfortunately not handling their permissions too. (Intent is to translate a system.posix_acl_access setxattr into corresponding permissions whenever possible.) Signed-off-by: Peter Seebach <peter.seebach@windriver.com>
In the fairly common case where someone is using setxattr() to specify the "posix_acl_access" attribute, but in fact the ACL list specified can be fully represented in a plain old mode, we intercept the request and just do a chmod. Even if the request can't be fully represented, we try to represent any aspects of it that we can in the plain old mode. Signed-off-by: Peter Seebach <peter.seebach@windriver.com>
The xattr first-pass implementation was allocating a buffer to hold the name and value for a set operation, then pseudo_client was allocating *another* buffer to hold the path and those two values. pseudo_client_op develops more nuanced argument handling, and also uses a static buffer for the extended paths it sometimes needs. So for the typical use case, only occasional operations will need to reallocate/expand the buffer, and we'll be down to copying things into that buffer once per operation, instead of having two alloc/free pairs and two copies. And of course, that wasn't two alloc/free pairs, it was one alloc/free pair and one alloc without a free. Whoops. Signed-off-by: Peter Seebach <peter.seebach@windriver.com>
Was using the length of the name instead of the length of the value on insert, but not on update, so initial settings of values were busted often. Signed-off-by: Peter Seebach <peter.seebach@windriver.com>
The "/* flags = AT_SYMLINK_NOFOLLOW */" comment only works if it comes AFTER the semicolon in wrapfuncs.in. Who knew? Fix those. Also rename the "flags" arguments for *setxattr() to "xflags" to avoid any confusion about the flags variable. Signed-off-by: Peter Seebach <peter.seebach@windriver.com>
The test case is based on the simple test of doing:
touch foo
getfattr -d foo
setfattr -n "user.dummy" -v "test" foo
getfattr -d foo
# file: foo
user.dummy="test"
setfattr -n "security.dummy" -v "test" foo
getfattr -n "security.dummy" foo
If pseudo is not running, the first part should work as long as extended
attributes are enabled, but the attempt to set "security...."
should result in a failure similar to:
setfattr: foo: Operation not permitted
As long as pseudo is working properly, no errors should be reported, and
the data should come back with the same values as were originally set.
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Clean-up: Allow specification of environment hints for subports scripts, such as whether xattr support is available. Also make configure guess at a bit width if none is specified. Signed-off-by: Peter Seebach <peter.seebach@windriver.com>
Turns out the checks for feature support were using plain cc,
not ${CC}, which could break tests. Also add a sanity check to the
xattr support to confirm that <attr/xattr.h> is available.
Signed-off-by: Peter Seebach <peter.seebach@windriver.com>
underlying fchmodat() will just fail, but GNU tar calls it that way anyway, figuring it'll just retry on failure, but we don't report the failure. Nor do we want to, because that's expensive and slow and will result in additional database round trips. But I don't want to fail out right away, so for now, just strip the flag.
Change the handling of fchmodat(AT_SYMLINK_NOFOLLOW) to reject it if the host system does, so we preserve host system behavior. Mask out group/other write bits when actually creating files to reduce risks to filesystem integrity.
Wait until the server has finished processing all of our messages before exiting. Otherwise, it's possible for a command which sends a no-response message and then exits to be followed by another command which assumes the first one's done, and the second command's messages can get processed first.
Various wrappers checked for a non-null pseudo_get_value("PSEUDO_UNLOAD") to
determine whether the environment should include the pseudo variables. None
of those checks freed the returned value when it was not null. The new
check function does.
The new check function also sees whether PSEUDO_UNLOAD was defined in the
environment that should be used in the wrapped system call. This allows
pkg_postinst scripts to strip out the LD_PRELOAD setting, for example before
invoking qemu to execute commands in an environment that does not have
libpseudo.so.
[YOCTO #4843]
Signed-off-by: Peter A. Bigot <pab@pabigot.com>
Signed-off-by: Peter Seebach <peter.seebach@windriver.com>
PSEUDO_DB_MODE restores a proposed mode's 0700 bits, but now that we're masking 022 out, it should also restore those. Change it to restore 0722 from a proposed mode before sending to database.
We used to rely on filesystem operations to apply the umask when appropriate, but when we started masking out 022, that stopped working. Start watching umask.
Also for lstat, but that probably never matters because in Linux you will never actually call lstat without working really hard at it, because you end up calling __lxstat anyway. (Was already doing the right thing for Darwin.)
strlen(array) isn't a constant expression, even though gcc can sometimes figure it out at compile time.
The sqlite flags don't need to be present if they don't have meaningful values. I think.
More complicated, because we actually need to make com.apple stuff work probably.
We don't want to pick up newer memcpy because pseudo sometimes has to run host binaries even when built against a newer libc.
The assumption that a host is either x86_64 or x86_32 does not hold well on target systems.
XFS apparently has 64-bit inodes. Our inode data path was *almost* 64-bit clean. This doesn't require a database format change because sqlite3 doesn't distinguish, but it will probably invalidate existing files.db things on XFS. But they were broken anyway.
It turns out that, in the fairly common case where the did-unlink stuff has saved us from worse problems, pseudo produces probably-spurious error messages about the path mismatch when the did-unlink shows up. Change that into a debug message. Also fix a typo in a comment.
So it turns out that the sanity checks should be skipped on did_unlink, because otherwise if an inode gets reused for a different file type, it'll get nuked. This is pretty rare, but appears to bite us occasionally during debug stripping.
Trying to track down problems which sometimes result in files showing up as nameless files, producing clashes later. Looks like there were two issues; one is we were creating links for files that we'd already found by inode. The other is that rename was sending bogus LINK messages in some cases. Also simplified the find_file_dev path to extract the path as part of the initial operation, since there wasn't any case where that wasn't being done immediately afterwards.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
this syncs the github account with version 1.6.2