v18.2.0-1
v18.2.0-1
2023-09-09 Bazaah github@luxolus.com
pkgbuild: pkgver=v18.2.0 pkgrel=1
2023-09-09 Bazaah github@luxolus.com
justfile: add cexec, ctest, repackage targets
- cexec
Executes the given arg list inside the chroot
> Build the 'osd' CMake target
`just cexec make -j$(nproc) -C build osd`
- ctest
Runs ctest (with the optional args) inside the chroot
> Run testsuite
`just ctest`
> Rerun only failed tests
`just ctest --retry-failed`
> Run a regex filtered list of tests
`just ctest -R some_unittest`
- repackage
Repackages the project without triggering a rebuild
2023-09-09 Bazaah github@luxolus.com
justfile: improvements to upload to handle new packages
Now use an @all alias instead of naming packages directly, and pretty
print the uploadees with column -t (from util-linux)
justfile: remove watch, logs
These never worked correctly anyway
2023-09-09 Bazaah github@luxolus.com
gh-upload-artifact: try pretty print happy path of upload
If curl returns success, assume the result is json and pipe through jq.
This improves the output substantially as before it printed a
compressed no-newline json blob for each artifact uploaded.
2023-09-09 Bazaah github@luxolus.com
pkgbuild: updates to install files for v18
Minor updates to the installed files for:
- ceph-common
- librbd
- librgw
- ceph-cephadm
2023-09-09 Bazaah github@luxolus.com
pkgbuild: enable optional rbd components
- WITH_RBD_RWL, WITH_RBD_SSD_CACHE
Enables the rbd writeback cache, both pmem and ssd variants.
- WITH_RBD_MIRROR
Enables mirroring rbd images between ceph clusters, similar to rgw
multi-zonal replication
These changes were prompted by github.com/Enelar in
github.com/bazaah/aur-ceph/pull/18/commits/41a3081df83fb7ab9fd9d94f736ded417afb58da
References: https://docs.ceph.com/en/reef/rbd/rbd-persistent-write-log-cache
References: https://docs.ceph.com/en/reef/rbd/rbd-mirroring/
2023-09-09 Bazaah github@luxolus.com
pkgbuild: disable proprietary rgw backends
These are some seagate/intel specific features, seemingly to use ceph
rgw as the S3 interface to their products, and/or supporting some intel
encrypt/decrypt hardware.
These are (currently) disabled by default, but I have added them for
future proofing.
I think these will never be valid/useful compile options on Archlinux.
2023-09-09 Bazaah github@luxolus.com
pkgbuild: disable tracing / opentelemetry
My reasoning here is to avoid adding _even more_ dependencies to
ceph-libs. I do want to revisit these when/if I break up the mega
packages into separate libs
2023-09-09 Bazaah github@luxolus.com
pkgbuild: reenable system utf8proc
Was switched to default disabled, reenabling it here for our builds.
Upstream-Ref: https://github.com/ceph/ceph/commit/90662cad562fffbdeac33f3b79eac6a02eff8c2a
2023-09-09 Bazaah github@luxolus.com
pkgbuild: use backport-log-runway-expansion.patch
2023-09-09 Bazaah github@luxolus.com
repo: add backport-log-runway-expansion.patch
This fixes failures found in
test_concurrent_dir_link_and_compact_log_56210 where the underlying
machinery would attempt to write into a too-small ondisk segment.
We just backport the entire patch from `main`.
Upstream-ref: https://github.com/ceph/ceph/commit/93961385d79a9fef44266181ce6a2c2b805093d4
References: https://github.com/ceph/ceph/blob/main/src/test/objectstore/test_bluefs.cc#L1359
References: https://github.com/ceph/ceph/pull/50216
References: https://tracker.ceph.com/issues/58759
2023-09-09 Bazaah github@luxolus.com
pkgbuild: use fix-ecode-shec-test.patch
2023-09-09 Bazaah github@luxolus.com
repo: add ceph-18.2.0-fix-ecode-shec-test.patch
Fixes some undefined behavior in
src/test/erasure-code/TestErasureCodeShec_arguments.cc found while
when attempting to cast std::initializer_list to std::array.
Instead, use std::vector which has a constructor for
std::initializer_list<T>.
References: https://en.cppreference.com/w/cpp/container/array/to_array
2023-09-09 Bazaah github@luxolus.com
pkgbuild: use ceph-18.2.0-fmt10-fixes.patch
2023-09-09 Bazaah github@luxolus.com
repo: add ceph-18.2.0-fmt10-fixes.patch
This is fairly wide ranging series of patches, fixing multiple compile
errors owing changes in fmtlib 9 and 10.
Care was taken to attempt to not introduce #includes for fmtlib headers
in the public headers of the project, largely through defining a sister
<header file>_fmt.h header that just contains the necessary fmtlib
definitions, along with importing these files into src/... files as
needed. This appears to be the convention (at least, in some parts) of
the upstream codebase.
Briefly outlined, these are issues fixed:
1. Missing fmt::ostream_formatter definitions for types
These used to be implicit, but now fmtlib requires that types
specifically opt into this behavior.
~ src/common/LogEntry.h: clog_type, LogEntry
+ src/include/neorados/RADOS_fmt.hpp: neorados::Object
+ src/include/types_fmt.h: shard_id_t
2. Missing format_to() interface for ceph_le<T>
According to the upstream docs, it should be possible to just use the
explicit cast to the underlying T for formatting, but for whatever
reason, this doesn't work.
Therefore, define a templated format_to() for ceph_le<T> casting to
the underlying T, explicitly.
This is a fmtlib 10 feature -- previously format_to() was reserved for
enums. However, this is the simplest method I can see to avoid needing
to #include fmtlib headers in this header tree.
~ src/include/byteorder.h: ceph_le<T>
3. Make fmt::formatter.format definitions const
Required since fmtlib 8, but seemingly only became an issue in 10.
~ src/osd/osd_types_fmt.h: chunk_info_t, pg_info_t, SnapSet,
ScrubMap::object, ScrubMap
4. Explicitly from snapid_t to uint64_t for fmt::sprintf calls
As the fmt::formatter.parse implementation for snapid_t does not
handle the formatting qualifier used (or any at all).
I think previously this implicitly was casted to uint64_t anyway, so
we're just making the behavior explicit.
~ src/osd/SnapMapper.cc: snapid_t
5. Adding a fmt::formatter implementation for EntityName
Appears to have another casualty in the "no longer try implicit casts"
behavior change of fmtlib 10
~ src/common/LogEntry.h: EntityName
In addition to the above, we defer LogEntry's formatter to the ostream
implementation rather than duplicating the same output, separately in
the fmt::formatter.format definition.
References: https://github.com/fmtlib/fmt/releases/tag/10.0.0
References: https://github.com/fmtlib/fmt/releases/tag/8.0.0
2023-09-09 Bazaah github@luxolus.com
repo: update ceph-17.2.4-test-bluefs-split.patch
mostly fixes to line offsets, but there was one real if small change to
a variable init.
2023-09-09 Bazaah github@luxolus.com
repo: rm upstreamed iterator depreciations
This commit removes the patches to three files from
ceph-17.2.5-fix-iterator-depreciations:
- src/include/xlist.h
- src/include/rados/librados.hpp
- src/crimson/os/seastore/omap_manager/btree/string_kv_node_layout.h
As the upstream has fixed them.
However, the patches to the following have not yet been applied
upstream, and thus remain:
- src/include/rangeset.h
- src/msg/async/dpdk/circular_buffer.h
Upstream-Ref: https://github.com/ceph/ceph/commit/77a3ed56a7b7326814ea5d37214bec1d88ec334e
Upstream-Ref: https://github.com/ceph/ceph/commit/691119c389e119a03aea606b7c4093d46c6db43d
Upstream-Ref: https://github.com/ceph/ceph/commit/905969611d256b6613bd86347bd51ddeb0603b79
Upstream-Ref: https://github.com/ceph/ceph/commit/c662ee698c6366e1b53856eb1a2c903c301a426f
2023-09-09 Bazaah github@luxolus.com
pkgbuild: update tox-mypy-false-postive.patch
same patch as before, but context needs updating to apply correctly
2023-09-09 Bazaah github@luxolus.com
pkgbuild: rm upstreamed gcc13 patches
The changes within are now included in the upstream, and are no longer
needed
Upstream-ref: https://github.com/ceph/ceph/pull/50438
2023-09-09 Bazaah github@luxolus.com
pkgbuild: rm delete-test-librados-asio.patch
No longer applies to the upstream
2023-09-09 Bazaah github@luxolus.com
pkgbuild: rm -DFMT_DEPRECATED_OSTREAM
There seems to just be a large amount of effort across the entire
project for this one. Not surprising, given my investigations but it
does make it difficult to provide a singular upstream reference for this
change.
Instead I'll just link directly to all the commits involved since
v17.2.0...
- ceph/ceph@d8d93b0fe3b crimson/osd: correct the fmt::formatter<WatchTimeoutRequest>
- ceph/ceph@71b9344f299 crimson/os: specialize fmt::formatter<> for StagedIterator
- ceph/ceph@32e207da3f3 crimson/os: include fmt/os.h for printing std::error_code
- ceph/ceph@85830863fa4 crimson: specialize fmt::formatter<> for formatting using {fmt}
- ceph/ceph@76e4c5d83c2 test/librados/aio_cxx: use fmt formatting
- ceph/ceph@2e7aea7b0fd crimson/os: print pointer using fmt::ptr()
- ceph/ceph@c9569f8eb02 crimson: specialize fmt::formatter<>
- ceph/ceph@01f6bef6e52 *: specialize fmt::formatter<> for formatting using {fmt}
- ceph/ceph@3f67d67ede1 crimson: specialize fmt::formatter<> to print using {fmt}
- ceph/ceph@42dc970392e crimson/net: print shared_ptr using fmt::ptr()
- ceph/ceph@fe1e6b6c4eb neorados: compile with fmt v9
- ceph/ceph@b2c61113ffb crimson/net: define fmt::formatter for write_state_t
- ceph/ceph@1f1398bc515 mon: Fix snprintf truncation warning by switching to fmtlib
- ceph/ceph@93dfb7e88b9 common: improving fmtlib handling of ceph::utime_t
- ceph/ceph@3d7a895d197 include/object_fmt: mark fmt::formatter<>::format() const
- ceph/ceph@2901943618f common/LogEntry: specialize fmt::formatter<LogEntry>
- ceph/ceph@6e223a68021 common/Journald: include msg/msg_fmt.h
- ceph/ceph@c0a3d460297 mon: use fmt::format for stderr cluster logging
- ceph/ceph@cef04bc4ed0 common: use fmt::format for stderr logging
- ceph/ceph@31aa539f266 osd: use fmt/ranges.h for printing ranges
- ceph/ceph@b7b7ff4cce3 include/types_fmt: use fmt::join() to format containers
- ceph/ceph@b287fadde3e fmt: pickup v8.1.1
- ceph/ceph@3c46dd64bbc cmake: bump up required fmt version
- ceph/ceph@01f9dc84826 msg/msg_type: use fmt::format to print entity_addr
- ceph/ceph@ac114ad172b crimson/os: use fmt::format() instead of sstream
- ceph/ceph@bb9d099ede8 crimson: specialize fmt::formatter<>() for crimson types
- ceph/ceph@3fd91d7e6d9 crimson/os/seastore: drop compile-time fmt validation
2023-09-09 Bazaah github@luxolus.com
pkgbuild: rm ceph-17.2.4.-tox-cephadm-rm.patch
Underlying issue found in f5af709 has been fixed upstream, re-enabling
test.
Upstream-Ref: https://github.com/ceph/ceph/commit/b5218528d63ae87ca77229bce1635c62411959e8
2023-09-09 Bazaah github@luxolus.com
repo: revert backport-with-fmt-version.patch
This was a temporary fix for a v17 rebuild. The upstream has fixed
many, many fmt issues in the new release.
Reverts: 7dfdbe1
Reverts: 34d11c3
2023-09-09 Bazaah github@luxolus.com
pkgbuild: add inetutils,xmlstarlet to checkdepends
As the check_generated.sh test now depends on xmlstarlet, and
`hostname` is used a bunch.
pkgbuild: add makedepend 'thrift'
required by the upstream now
pkgbuild: cleanup pkgname, reorganize arch,pkgdesc
Mostly updating the pkgname array to reflect the refactoring done on
packaging, and a slight change in field ordering to move pkgdesc higher
in the file
pkgbuild: reorganize makedepends, checkdepends
Improve formatting, and breakup into groups (system and python,
currently), so as to escape the giant word soup that currently exists.
2023-09-09 Bazaah github@luxolus.com
pkgbuild: package virtual targets
These are virtual (dependency only) packages that serve a couple
purposes:
1. Backwards compatibility with the existing Archlinux ceph packages.
That is:
- ceph-libs: all non-binary packages
- ceph-mgr: (same as before / not virtual)
- ceph: all binary packages
2. Stable targets for certain functionality, namely cluster components
and the `ceph` CLI utility which is used for most interactions with
a cluster.
- ceph-libs
- ceph-cluster
- ceph-cli
- ceph
2023-09-09 Bazaah github@luxolus.com
pkgbuild: package misc. ceph tooling / libs
These are mostly used by the upstream development team for interacting
with various internals of the cluster and live tests.
However, the impetus for this entire patchset also lurks: java-cephfs.
A package that requires we pull in the entire java runtime, likely for a
package absolutely no one uses. Good riddance.
- ceph-tools
- ceph-test
- java-cephfs
2023-09-09 Bazaah github@luxolus.com
pkgbuild: package more client / utils
These are a couple useful utilities for interfacing with ceph, of which
ceph-volume is likely the most important, as is it responsible for
provisioning devices for use as OSDs.
cephadm is an honorable mention, being a 9k line python file used in
managing clusters... if you like the idea of having 9k lines of python
sitting between you and your entire storage backplane.
(I do not)
- ceph-volume
- cephfs-shell
- cephfs-top
- ceph-cephadm
2023-09-07 Bazaah github@luxolus.com
pkgbuild: package ceph cluster components
These comprise the daemons and associated tooling that form the cluster
or "backend" of a ceph deployment.
Notably, they have a dependencies on both the core libs, python libs
_and_ librgw (a client), as despite being a backend or server component;
ceph-rgw is _actually_ a client of ceph (and librgw specifically), that
just so happens to run as a daemon -- providing object storage.
- ceph-base
- ceph-mon
- ceph-mgr
- ceph-osd
- ceph-mds
- ceph-rgw
2023-09-07 Bazaah github@luxolus.com
pkgbuild: package ceph python libs
These are python bindings to the underlying C/C++ libs, and are used for
most of the ancillary CLI tooling ceph provides, including as support
tooling for cluster components.
- python-ceph-common
- python-rados
- python-rbd
- python-cephfs
- python-rgw
2023-09-07 Bazaah github@luxolus.com
pkgbuild: package core clients
These form the core clients for ceph; rbd (block), cephfs
(filesystem), rgw (object).
- librbd
- ceph-rbd
- libcephfs
- ceph-cephfs
- librgw
2023-09-07 Bazaah github@luxolus.com
pkgbuild: package core libs
These are the core / common libraries that almost everything else will
depend on at some level.
libcephsqlite is an outlier, but is also depended on by other ceph
components despite _technically_ being a RADOS client, hence its
inclusion.
- ceph-compressor
- ceph-crypto
- ceph-erasure
- ceph-common
- librados
- ceph-rados
- libcephsqlite
2023-09-07 Bazaah github@luxolus.com
pkgbuild: add func _make_ceph_packages
The true workhorse for packaging ceph, this function is responsible for
calling make install, and _ensuring all files accounted for_ -- moved or
deleted -- before yielding.
It will contain a series of `_package` invocations, one per package,
with each actual package function being a thin wrapper around:
'mv __pkg__/$pkgname/* "$pkgdir"'
2023-09-07 Bazaah github@luxolus.com
pkgbuild: add _package, _print funcs
These are the beginning of a refactoring of how we handle the package_*
functions.
Currently, we run `make install` once per packaging function and attempt
to delete the bits that aren't related to the package currently running.
This has several downsides, including conflicting file owners, missed
addition(al file)s in later versions and speed.
These new functions will serve as the engine for enumerating the precise
file dependencies for each package, and provide a pretty output during
the packaging process of this list.
2023-09-07 Bazaah github@luxolus.com
pkgbuild: rm old package() functions