Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Simplify and cleanup the binaries and libraries copying code. #1521

Merged
merged 4 commits into from Oct 4, 2017
Merged

Simplify and cleanup the binaries and libraries copying code. #1521

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
3973399
Simplify and cleanup the binaries and libraries copying code
jsmeix Sep 29, 2017
efc53aa
More explanatory comments and using 'readlink -m' because no componen…
jsmeix Oct 4, 2017
dc44fb4
Replaced actually local variable 'BINARIES' with 'all_binbaries'
jsmeix Oct 4, 2017
6f364bb
Merge branch 'master' into also_find_libraries_required_by_libraries_…
jsmeix Oct 4, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
93 changes: 56 additions & 37 deletions usr/share/rear/build/GNU/Linux/100_copy_as_is.sh
View file
Open in desktop
@@ -1,53 +1,72 @@
# 400_copy_as_is.sh
#
# copy files and directories that should be copied over as-is to the rescue
# systems. Checks also for library dependencies of executables and adds
# them to the LIBS list, if they are not included in the copied files.
# Copy files and directories that should be copied as-is into the recovery system.
# Check also for library dependencies of executables in all the copied files and
# add them to the LIBS list if they are not yet included in the copied files.

LogPrint "Copying files and directories"
Log "Files being copied: ${COPY_AS_IS[@]}"
Log "Files being excluded: ${COPY_AS_IS_EXCLUDE[@]}"

for f in "${COPY_AS_IS_EXCLUDE[@]}" ; do echo "$f" ; done >$TMP_DIR/copy-as-is-exclude
tar -v -X $TMP_DIR/copy-as-is-exclude \
-P -C / -c "${COPY_AS_IS[@]}" 2>$TMP_DIR/copy-as-is-filelist | \
tar $v -C $ROOTFS_DIR/ -x >/dev/null
StopIfError "Could not copy files and directories"
Log "Finished copying COPY_AS_IS"

# fix ReaR directory if running from checkout
if [[ "$REAR_DIR_PREFIX" ]] ; then
for dir in /usr/share/rear /var/lib/rear ; do
ln $v -sf $REAR_DIR_PREFIX$dir $ROOTFS_DIR$dir>/dev/null
local copy_as_is_filelist_file="$TMP_DIR/copy-as-is-filelist"
local copy_as_is_exclude_file="$TMP_DIR/copy-as-is-exclude"

# Build the list of files and directories that are excluded from being copied:
local excluded_file=""
for excluded_file in "${COPY_AS_IS_EXCLUDE[@]}" ; do
echo "$excluded_file"
done >$copy_as_is_exclude_file

# Copy files and directories as-is into the recovery system except the excluded ones and
# remember what files and directories were actually copied in a copy_as_is_filelist_file
# which is the reason that the first 'tar' must be run in verbose mode:
if ! tar -v -X $copy_as_is_exclude_file -P -C / -c "${COPY_AS_IS[@]}" 2>$copy_as_is_filelist_file | tar $v -C $ROOTFS_DIR/ -x 1>/dev/null ; then
Error "Failed to copy files and directories in COPY_AS_IS minus COPY_AS_IS_EXCLUDE"
fi
Log "Finished copying files and directories in COPY_AS_IS minus COPY_AS_IS_EXCLUDE"

# Build an array of the actual regular files that are executable in all the copied files:
local copy_as_is_executables=()
local copy_as_is_file=""
while read -r copy_as_is_file ; do
# Skip non-regular files (like directories and device files):
test -f "$copy_as_is_file" || continue
# Skip symbolic links (only care about symbolic link targets):
test -L "$copy_as_is_file" && continue
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @jsmeix , do you know why to skip symbolic links? The previous versions did not have this (the test was if [[ ! -d "$REPLY" && -x "$REPLY" ]]; then, which includes symlinks).

I believe skipping symlinks is wrong: if the symlink is dangling in the recovery system, the target will be eventually copied by build/default/490_fix_broken_links.sh, but without libraries, and verification of the recovery system will fail and abort rear mkrescue.
(If the symlink is not dangling, there will be duplicates in the library list if we don't skip it, but I suppose that's not a big problem?)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removing this line does not seem to have any ill effect in a quick test

Copy link
Member Author

@jsmeix jsmeix Nov 2, 2023 •
edited

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@pcahyna
could you please make a new GitHub issue for it
where you explain how things went wrong for you
so I could better understand the whole thing
and we could better discuss it
(I do no longer trust comments in pull requests).

I don't remember why I skip symlinks there.
The initial comment of this code part

# Build an array of the actual regular files that are executable in all the copied files:

also only tells that symlinks are not considered
but it does not tell why.

I assume your failing case is a rather special case
because I committed that change on Sep 29 2017 as
3973399
and I merged its pull request on Oct 4 2017
so since 6 years it had worked without issues
(at least without reported issues as far as I know)
which is why I like to fully understand the root cause
behind your current issue why it now fails for you
before I "just change" any symlink behaviour in ReaR
because in practice symlinks are a hell of endless pain
(not only in ReaR but everywhere).

For the (not so) fun of it:
In the end symlinks are just one implementation of
RFC 1925 item 6a
"It is always possible to add another level of indirection"
which is (in my experience) a root cause behind 90% of issues.
In my experience another root cause behind 90% of issues
is RFC 1925 item 5 "aglutenate [sic] multiple separate
problems into a single complex interdependent solution"
so about 81% of issues have RFC 1925 items 5 and 6a
as root causes :-(

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jsmeix , #3064 . It also explains why this problem does not arise very often.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@pcahyna
thank you!

I will have a look tomorrow...

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After quick reading #3064 (comment)
I can now imagine even a possible reason why COPY_AS_IS
does not care about symlinks like PROGS does:
Because COPY_AS_IS means "copy as is but nothing more"?
This is only an offhanded guess.

# Remember actual regular files that are executable:
test -x "$copy_as_is_file" && copy_as_is_executables=( "${copy_as_is_executables[@]}" "$copy_as_is_file" )
done <$copy_as_is_filelist_file
Log "copy_as_is_executables = ${copy_as_is_executables[@]}"

# Check for library dependencies of executables in all the copied files and
# add them to the LIBS list if they are not yet included in the copied files:
Log "Adding required libraries of executables in all the copied files to LIBS"
local required_library=""
for required_library in $( RequiredSharedOjects "${copy_as_is_executables[@]}" ) ; do
# Skip when the required library was already actually copied by 'tar' above:
grep -q "$required_library" $copy_as_is_filelist_file && continue
# Skip when the required library is already in LIBS:
IsInArray "$required_library" "${LIBS[@]}" && continue
Log "Adding required library '$required_library' to LIBS"
LIBS=( "${LIBS[@]}" "$required_library" )
done
Log "LIBS = ${LIBS[@]}"

# Fix ReaR directories when running from checkout:
if test "$REAR_DIR_PREFIX" ; then
Log "Fixing ReaR directories when running from checkout"
local rear_dir=""
for rear_dir in /usr/share/rear /var/lib/rear ; do
ln $v -sf $REAR_DIR_PREFIX$rear_dir $ROOTFS_DIR$rear_dir 1>/dev/null
done
fi

### Copy configuration directory
Log "Copying ReaR configuration directory"
# Copy ReaR configuration directory:
mkdir $v -p $ROOTFS_DIR/etc/rear
# This will do same job as lines below.
# On top of that, it does not throw log warning like:
# "cp: missing destination file operand after"
# if hidden file (.<filename>) is missing in $CONFIG_DIR
cp $v -r $CONFIG_DIR/. $ROOTFS_DIR/etc/rear/ >&2

COPY_AS_IS_EXELIST=()
while read -r ; do
if [[ ! -d "$REPLY" && -x "$REPLY" ]]; then
COPY_AS_IS_EXELIST=( "${COPY_AS_IS_EXELIST[@]}" "$REPLY" )
fi
done <$TMP_DIR/copy-as-is-filelist
Log "COPY_AS_IS_EXELIST = ${COPY_AS_IS_EXELIST[@]}"

Log "Adding required libraries to LIBS with checking COPY_AS_IS_EXELIST"
# add required libraries to LIBS, skip libraries that are part of the copied files.
while read -r ; do
lib="$REPLY"
if ! IsInArray "$lib" "${COPY_AS_IS_EXELIST[@]}"; then
# if $lib is NOT part of the copy-as-is fileset, then add it to the global libs
LIBS=( ${LIBS[@]} $lib )
else
Log "Not adding $lib to LIBS because it is already in COPY_AS_IS_EXELIST"
fi
done < <( SharedObjectFiles "${COPY_AS_IS_EXELIST[@]}" | sed -e 's#^#/#' )
Log "LIBS = ${LIBS[@]}"
cp $v -r $CONFIG_DIR/. $ROOTFS_DIR/etc/rear/ 1>&2

72 changes: 47 additions & 25 deletions usr/share/rear/build/GNU/Linux/390_copy_binaries_libraries.sh
View file
Open in desktop
@@ -1,51 +1,73 @@
# 200_copy_binaries_libraries.sh
#
# copy binaries and libraries for Relax-and-Recover
# Copy binaries and libraries for Relax-and-Recover.
#
# This file is part of Relax-and-Recover, licensed under the GNU General
# Public License. Refer to the included COPYING for full text of license.

LogPrint "Copying binaries and libraries"

# calculate binaries from needed progs
# Calculate binaries from needed progs:
Log "Determining binaries from PROGS and REQUIRED_PROGS"
declare -a BINARIES=( $( for bin in "${PROGS[@]}" "${REQUIRED_PROGS[@]}" ; do
file="$( get_path "$bin" )"
if [[ -x "$file" ]] ; then
echo $file
Log "Found binary $file"
fi
done | sort -u ) )
local bin=""
local bin_path=""
local all_binaries=( $( for bin in "${PROGS[@]}" "${REQUIRED_PROGS[@]}" ; do
bin_path="$( get_path "$bin" )"
if test -x "$bin_path" ; then
echo $bin_path
Log "Found binary $bin_path"
fi
done | sort -u ) )

# copy binaries
Log "Binaries being copied: ${BINARIES[@]}"
BinCopyTo "$ROOTFS_DIR/bin" "${BINARIES[@]}" >&2 || Error "Failed to copy binaries"
# Copy binaries:
Log "Binaries being copied: ${all_binaries[@]}"
BinCopyTo "$ROOTFS_DIR/bin" "${all_binaries[@]}" 1>&2 || Error "Failed to copy binaries"

# copy libraries
declare -a all_libs=( $( for lib in ${LIBS[@]} $( SharedObjectFiles "${LIBS[@]}" "${BINARIES[@]}" | sed -e 's#^#/#' ) ; do
echo $lib
done | sort -u ) )
# Copy libraries:
# It is crucial to also have all LIBS itself in all_libs because RequiredSharedOjects()
# outputs only those libraries that are required by a library but not the library itself
# so that without all LIBS itself in all_libs those libraries in LIBS are missing that
# are not needed by a binary in all_binaries (all_binaries were already copied above).
# RequiredSharedOjects outputs the required shared objects on STDOUT.
# The output are absolute paths to the required shared objects.
# The output can also be symbolic links (also as absolute paths).
# In case of symbolic links only the link but not the link target is output.
# Therefore for symbolic links also the link target gets copied below.
local all_libs=( "${LIBS[@]}" $( RequiredSharedOjects "${all_binaries[@]}" "${LIBS[@]}" ) )

function ensure_dir() {
local dir=${1%/*}
test -d $ROOTFS_DIR$dir || mkdir $v -p $ROOTFS_DIR$dir >&2
test -d $ROOTFS_DIR/$dir || mkdir $v -p $ROOTFS_DIR/$dir 1>&2
}

function copy_lib() {
local lib=$1
ensure_dir $lib
test -e $ROOTFS_DIR/$lib || cp $v -a -f $lib $ROOTFS_DIR$lib >&2
test -e $ROOTFS_DIR/$lib || cp $v -a -f $lib $ROOTFS_DIR/$lib 1>&2
}

Log "Libraries being copied: ${all_libs[@]}"
local lib=""
local link_target=""
for lib in "${all_libs[@]}" ; do
if [[ -L $lib ]] ; then
target=$( readlink -f $lib )
copy_lib $target
ensure_dir $lib
ln $v -sf $target $ROOTFS_DIR$lib >&2
if test -L $lib ; then
# None of the link target components may already exist when 'readlink' is called
# because they could be first created by the subsequent 'copy_lib $link_target'
# so that 'readlink -m' must be used:
link_target=$( readlink -m $lib )
if test "$link_target" ; then
copy_lib $link_target || LogPrintError "Failed to copy symlink target '$link_target'"
# If in the original system there was a chain of symbolic links like
# /some/path/to/libfoo.so.1 -> /another/path/to/libfoo.so.1.2 -> /final/path/to/libfoo.so.1.2.3
# it gets simplified in the recovery system to
# /some/path/to/libfoo.so.1 -> /final/path/to/libfoo.so.1.2.3
ensure_dir $lib || LogPrintError "Failed to create directories of symlink '$lib'"
ln $v -sf $link_target $ROOTFS_DIR/$lib 1>&2 || LogPrintError "Failed to link '$link_target' as symlink '$lib'"
else
LogPrintError "Cannot copy symlink '$lib' because it has no link target"
fi
else
copy_lib $lib
copy_lib $lib || LogPrintError "Failed to copy '$lib'"
fi
done

Expand All @@ -61,5 +83,5 @@ done
# even an inconsistent libraries configuration works sufficiently, for example see
# https://github.com/rear/rear/issues/772
# TODO: Get the libraries configuration in the recovery system consistent in any case.
ldconfig $v -r "$ROOTFS_DIR" 1>&2 || LogPrintError "Configuring rescue/recovery system libraries with ldconfig failed which may casuse arbitrary failures"
ldconfig $v -r "$ROOTFS_DIR" 1>&2 || LogPrintError "ldconfig failed to configure rescue/recovery system libraries which may casuse arbitrary failures"

121 changes: 70 additions & 51 deletions usr/share/rear/lib/linux-functions.sh
View file
Open in desktop
Expand Up @@ -114,57 +114,76 @@ function BinCopyTo () {
done
}

# Resolve dynamic library dependencies. Returns a list of symbolic links
# to shared objects and shared object files for the binaries in $@.
# This is the function copied from mkinitrd off SUSE 9.3
function SharedObjectFiles () {
has_binary ldd || Error "SharedObjectFiles failed because there is no ldd binary"

# Default ldd output (when providing more than one argument) has 5 cases:
# 1. Line: "file:" -> file argument
# 2. Line: " lib => (mem-addr)" -> virtual library
# 3. Line: " lib => not found" -> print error to stderr
# 4. Line: " lib => /path/lib (mem-addr)" -> print $3
# 5. Line: " /path/lib (mem-addr)" -> print $1
local -a initrd_libs=( $( ldd "$@" | awk '
/^\t.+ => not found/ { print "WARNING: Dynamic library " $1 " not found" > "/dev/stderr" }
/^\t.+ => \// { print $3 }
/^\t\// { print $1 }
' | sort -u ) )

### FIXME: Is this still relevant today ? If so, make it more specific !

# Evil hack: On some systems we have generic as well as optimized
# libraries, but the optimized libraries may not work with all
# kernel versions (e.g., the NPTL glibc libraries don't work with
# a 2.4 kernel). Use the generic versions of the libraries in the
# initrd (and guess the name).
# local lib= n= optimized=
# for ((n=0; $n<${#initrd_libs[@]}; n++)); do
# lib=${initrd_libs[$n]}
# optimized="$(echo "$lib" | sed -e 's:.*/\([^/]\+/\)[^/]\+$:\1:')"
# lib=${lib/$optimized/}
# if [ "${optimized:0:3}" != "lib" -a -f "$lib" ]; then
# #echo "[Using $lib instead of ${initrd_libs[$n]}]" >&2
# initrd_libs[$n]="${lib/$optimized/}"
# fi
# echo Deoptimizing "$lib" >&2
# done

local lib="" link=""
for lib in "${initrd_libs[@]}" ; do
lib="${lib:1}"
while [ -L "/$lib" ] ; do
echo $lib
link="$( readlink "/$lib" )"
case "$link" in
(/*) lib="${link:1}" ;;
(*) lib="${lib%/*}/$link" ;;
esac
done
echo $lib
echo $lib >&2
done | sort -u
# Determine all required shared objects (shared/dynamic libraries)
# for programs and/or shared objects (binaries) specified in $@.
# RequiredSharedOjects outputs the required shared objects on STDOUT.
# The output are absolute paths to the required shared objects.
# The output can also be symbolic links (also as absolute paths).
# In case of symbolic links only the link but not the link target is output.
function RequiredSharedOjects () {
has_binary ldd || Error "Cannot run RequiredSharedOjects() because there is no ldd binary"
Log "RequiredSharedOjects: Determining required shared objects"
# It uses 'ldd' to determine all required shared objects because 'ldd' outputs
# also transitively required shared objects i.e. libraries needed by libraries,
# e.g. for /usr/sbin/parted also the libraries needed by the libparted library:
# # ldd /usr/sbin/parted
# linux-vdso.so.1 (0x00007ffd68fe1000)
# libparted.so.2 => /usr/lib64/libparted.so.2 (0x00007f0c72bee000)
# libtinfo.so.6 => /lib64/libtinfo.so.6 (0x00007f0c729c4000)
# libreadline.so.6 => /lib64/libreadline.so.6 (0x00007f0c72778000)
# libc.so.6 => /lib64/libc.so.6 (0x00007f0c723d5000)
# libuuid.so.1 => /usr/lib64/libuuid.so.1 (0x00007f0c721d0000)
# libdevmapper.so.1.02 => /lib64/libdevmapper.so.1.02 (0x00007f0c71f85000)
# libblkid.so.1 => /usr/lib64/libblkid.so.1 (0x00007f0c71d43000)
# libtinfo.so.5 => /lib64/libtinfo.so.5 (0x00007f0c71b0f000)
# /lib64/ld-linux-x86-64.so.2 (0x000055eff2882000)
# libselinux.so.1 => /lib64/libselinux.so.1 (0x00007f0c718e8000)
# libudev.so.1 => /usr/lib64/libudev.so.1 (0x00007f0c716c8000)
# libpthread.so.0 => /lib64/noelision/libpthread.so.0 (0x00007f0c714ab000)
# libpcre.so.1 => /usr/lib64/libpcre.so.1 (0x00007f0c71244000)
# libdl.so.2 => /lib64/libdl.so.2 (0x00007f0c71040000)
# libcap.so.2 => /lib64/libcap.so.2 (0x00007f0c70e3b000)
# librt.so.1 => /lib64/librt.so.1 (0x00007f0c70c32000)
# libm.so.6 => /lib64/libm.so.6 (0x00007f0c70935000)
# libresolv.so.2 => /lib64/libresolv.so.2 (0x00007f0c7071e000)
# # file /usr/lib64/libparted.so.2
# /usr/lib64/libparted.so.2: symbolic link to `libparted.so.2.0.0'
# # mv /usr/lib64/libparted.so.2.0.0 /usr/lib64/libparted.so.2.0.0.away
# # ldd /usr/sbin/parted /usr/lib64/libparted.so.2.0.0.away
# /usr/sbin/parted:
# linux-vdso.so.1 (0x00007ffc38505000)
# libparted.so.2 => not found
# libtinfo.so.6 => /lib64/libtinfo.so.6 (0x00007fe0f4b5e000)
# libreadline.so.6 => /lib64/libreadline.so.6 (0x00007fe0f4913000)
# libc.so.6 => /lib64/libc.so.6 (0x00007fe0f4570000)
# libtinfo.so.5 => /lib64/libtinfo.so.5 (0x00007fe0f433b000)
# /lib64/ld-linux-x86-64.so.2 (0x000055e2549e2000)
# /usr/lib64/libparted.so.2.0.0.away:
# linux-vdso.so.1 (0x00007fffdbb8f000)
# libuuid.so.1 => /usr/lib64/libuuid.so.1 (0x00007f3c9a87d000)
# libdevmapper.so.1.02 => /lib64/libdevmapper.so.1.02 (0x00007f3c9a633000)
# libblkid.so.1 => /usr/lib64/libblkid.so.1 (0x00007f3c9a3f0000)
# libc.so.6 => /lib64/libc.so.6 (0x00007f3c9a04d000)
# /lib64/ld-linux-x86-64.so.2 (0x0000563ffc5f1000)
# libselinux.so.1 => /lib64/libselinux.so.1 (0x00007f3c99e27000)
# libudev.so.1 => /usr/lib64/libudev.so.1 (0x00007f3c99c06000)
# libpthread.so.0 => /lib64/noelision/libpthread.so.0 (0x00007f3c999e9000)
# libpcre.so.1 => /usr/lib64/libpcre.so.1 (0x00007f3c99783000)
# libdl.so.2 => /lib64/libdl.so.2 (0x00007f3c9957e000)
# libcap.so.2 => /lib64/libcap.so.2 (0x00007f3c99379000)
# librt.so.1 => /lib64/librt.so.1 (0x00007f3c99171000)
# libm.so.6 => /lib64/libm.so.6 (0x00007f3c98e73000)
# libresolv.so.2 => /lib64/libresolv.so.2 (0x00007f3c98c5c000)
# The 'ldd' output (when providing more than one argument) has 5 cases.
# So we have to distinguish lines of the following form (indentation is done with tab '\t'):
# 1. Line: "/path/to/binary:" -> current file argument for ldd
# 2. Line: " lib (mem-addr)" -> virtual library
# 3. Line: " lib => not found" -> print error to stderr
# 4. Line: " lib => /path/to/lib (mem-addr)" -> print $3 '/path/to/lib'
# 5. Line: " /path/to/lib (mem-addr)" -> print $1 '/path/to/lib'
ldd "$@" | awk ' /^\t.+ => not found/ { print "Shared object " $1 " not found" > "/dev/stderr" }
/^\t.+ => \// { print $3 }
/^\t\// { print $1 } ' | sort -u
}

# Provide a shell, with custom exit-prompt and history
Expand Down