No reason to wait for make_writeable(); ensure we have a valid snapc
from the start.

Signed-off-by: Sage Weil <sage@redhat.com>

If REQUIRE_LUMINOUS is set on the OSDMap, put the SnapSet on the head
and make it a whiteout.  This is simpler and will eventually (once all
the old snapdir objects have been fixed up) let us remove a ton of
snapdir-related headaches.

This is surprisingly simple.  Everywhere else we work with snapdir is
already set up to handle the snapset on the head.  The only difference is
that we are preventing outselves from moving from the snapset-on-head
state to the snapset-on-snapdir-with-no-head state.  The only time this
happens is when the object is logically deleted in _delete_oid.

Signed-off-by: Sage Weil <sage@redhat.com>

We may still need to create a whiteout because clones still exist.

Arguably delete+ignore_cache is not the right way to remove whiteouts and
we should have a separate RADOS operation for this.  But we don't.

Signed-off-by: Sage Weil <sage@redhat.com>

Signed-off-by: Sage Weil <sage@redhat.com>

This will make it easier to identify users of this field that need to
conditionally use either the old legacy_snaps or the new
SnapSet::clone_snaps vector.

Signed-off-by: Sage Weil <sage@redhat.com>

Store the per-clone snaps list in SnapSet if (1) REQUIRE_LUMINOUS is set
in the OSDMap and (2) the SnapSet isn't a 'legacy' SnapSet that hasn't
been converted yet by scrub.

Signed-off-by: Sage Weil <sage@redhat.com>

During scrub, we assemble the clone legacy_snaps values and put them
in the SnapSet.

Note that we do not bother clearing legacy_snaps in the clones; this is
more complexity to do correctly and not worth the effort; we will always
have the SnapSet to indicate whether we need to look at the oldf ield
or not.

If the rebuild clone_snaps is not the correct size we bail out; this can
happen if some clones are missing oi or missing entirely, or if there are
extra clones.

Signed-off-by: Sage Weil <sage@redhat.com>

Signed-off-by: Sage Weil <sage@redhat.com>

- Assume that any snapset we update, if not require_luminous, is a net-new
legacy SnapSet.  (It might be an existing one, which would be net-0, but
that is harder to tell.)

Then, during scrub,

- Any unreadable oi is assumed to include a legacy snapset
- Any snapset we encounter if !require_luminous is legacy
- Any object that should have a snapset but doesn't (corrupt or missing)
is assumed to be legacy.
- If were trying to update a legacy Snapset but have to abort, then it is
still legacy.

We could assume that a missing/broken snapset is not legacy since it has
to be repaired anyway (and therefore shouldn't block upgrade), but I'm
not sure.  For now, we'll take the conservative approach of blocking the
upgrade if the snapset metadata is missing/corrupt.

Signed-off-by: Sage Weil <sage@redhat.com>

# Conflicts:
#	src/osd/PrimaryLogPG.cc

We include a check to make sure we do not delete a dirty whiteout if this
is a tier pool and the object is dirty.

Signed-off-by: Sage Weil <sage@redhat.com>

Signed-off-by: Sage Weil <sage@redhat.com>

Signed-off-by: Sage Weil <sage@redhat.com>

Signed-off-by: Sage Weil <sage@redhat.com>

Replicas need this in order to store the clones in SnapMapper.

Signed-off-by: Sage Weil <sage@redhat.com>

Signed-off-by: Sage Weil <sage@redhat.com>

Signed-off-by: Sage Weil <sage@redhat.com>

In the new world, head_exists is always true.  Stop clearing it in
_delete_oid (except for legacy compatibility), and fix the various callers
and assertions to compensate.  Note that we now use head_exists as a flag
to guide us into the snapdir branch of finish_ctx() (which will go away
post-luminous).

Signed-off-by: Sage Weil <sage@redhat.com>

- objects come in a different order, meh
- ss is on head, always, not snapdir.
- error messages on head, not snapdir

Signed-off-by: Sage Weil <sage@redhat.com>

If we encounter legacy snaps, add to snapmapper then; otherwise, use the
clone_snaps field in SnapSet to add all clones to snapmapper when we
process the head object.

Signed-off-by: Sage Weil <sage@redhat.com>

Do not touch the object before the put if we expect to see ENOENT.
We would get it anyway with older code, but with the snapset
changes we are on a whiteout and get ENODATA instead.

And really we shouldn't have been getting ENOENT after a touch
anyway!

Signed-off-by: Sage Weil <sage@redhat.com>

We no longer see dup entries on split.

Signed-off-by: Sage Weil <sage@redhat.com>

First, this is pointless--each test runs in a namespace so they don't
step on each other.  Second, leaving objects in place is an opportunity
for scrub to notice any issues we created.  Third, the cleanup asserts
that delete succeeds but if clones exist pgls will show whiteouts and then
delete will return ENOENT.  We could disable the assert, but why bother
even attempting a sloppy cleanup?

We need to preserve cleanup behavior for a few tests (notably the object
listing ones).

Signed-off-by: Sage Weil <sage@redhat.com>

Instead of deleting a pool, add a .NNN.DELETED suffix to the end.  This
keeps the data around long enough for it to be scrubbed later (in the
case of a teuthology job cleanup).

If you really want to delete the pool, then instead of the usual force
flag option you can pass --yes-i-really-really-mean-it-not-faking.  :)

Signed-off-by: Sage Weil <sage@redhat.com>

… pool even if faking

Signed-off-by: Sage Weil <sage@redhat.com>

This breaks the upgrade test from jewel.  We can probably revert it later.

Signed-off-by: Sage Weil <sage@redhat.com>

Signed-off-by: Sage Weil <sage@redhat.com>

Signed-off-by: Sage Weil <sage@redhat.com>

Signed-off-by: Sage Weil <sage@redhat.com>

Signed-off-by: Sage Weil <sage@redhat.com>

Signed-off-by: Sage Weil <sage@redhat.com>

…_finally

This lets us run multiple cleanup steps right before ceph
teardown.

Note that we drop the facet from multimon/ because it
doesn't factor out cluster creation before this step
properly.  That's fine because the require_luminous
cleanup shouldn't be related to the multimon tests.

Signed-off-by: Sage Weil <sage@redhat.com>

Signed-off-by: Sage Weil <sage@redhat.com>

We may be creating the head object to migrate the SnapSet to.

Signed-off-by: Sage Weil <sage@redhat.com>

Signed-off-by: Sage Weil <sage@redhat.com>

Signed-off-by: Sage Weil <sage@redhat.com>

 * Adds --with-blkin to autoconf (default without)
 * Adds BLKIN_LIBS to necessary automake files
 * Adds documentation for testing BlkKin tracing

Signed-off-by: Andrew Shewmaker <agshew@gmail.com>
Signed-off-by: Marios-Evaggelos Kogias <marioskogias@gmail.com>
Signed-off-by: Chendi.Xue <chendi.xue@intel.com>
Signed-off-by: Casey Bodley <cbodley@redhat.com>

Signed-off-by: Casey Bodley <cbodley@redhat.com>

zipkin_trace.h is a wrapper around ztracer.hpp, which provides a stub
implementation when WITH_BLKIN is not defined

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Avoids creating orphaned traces.

Signed-off-by: Victor Araujo <ve.ar91@gmail.com>

when set, Message::decode_trace() will always create a trace for
incoming messages, even if they didn't pass trace information

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Signed-off-by: Victor Araujo <ve.ar91@gmail.com>

This will enable passing trace information from rbd to librados for
aio_reads and aio_writes.

Signed-off-by: Victor Araujo <ve.ar91@gmail.com>

Signed-off-by: Sage Weil <sage@redhat.com>

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Signed-off-by: Sage Weil <sage@redhat.com>

Signed-off-by: Sage Weil <sage@redhat.com>

These are protocol features and cannot vary based on our compilation.
Encode and decode unconditionally.  The callers have already guarded these
field additions with a message version bump and are conditionally calling
decode_trace.

Signed-off-by: Sage Weil <sage@redhat.com>

- encode the same regardless of whether it is compiled in (!)
- encode in endian- and struct alignment-safe way.

Signed-off-by: Sage Weil <sage@redhat.com>

Signed-off-by: Sage Weil <sage@redhat.com>

Signed-off-by: Sage Weil <sage@redhat.com>

Signed-off-by: Sage Weil <sage@redhat.com>

Signed-off-by: Sage Weil <sage@redhat.com>

Signed-off-by: Sage Weil <sage@redhat.com>

We are unconditionally linking lttng-ust, which make selinux complain. We
should either

 - fix selinux rules and unconditionally link
 - dlopen at runtime based on an option (like we do for the current
   tracepoints)
 - ???

Signed-off-by: Sage Weil <sage@redhat.com>

I'm not entirely sure why we were ever signaling the cond here; the read
is synchronous!

Signed-off-by: Sage Weil <sage@redhat.com>

Thread 1 (_do_read)                 Thread 2 (_aio_thread)
queues aio
ioc->aio_wait()
  locks ioc->lock
  num_waiting++
                                    finishes aios
                                    ioc->aio_wake
                                      if (num_waiting)
                                        lock ioc->lock (blocks)
  num_running == 0, continue
  ioc->unlock
                                        ioc->lock taken
do_read destroys ioc
                                        use after free, may block forever

The last bit of the race may vary; the key thing is that thread 2 is
taking a lock that thread 1 can destroy; thread 2 doesn't have it pinned
in memory.

Fix this by simplifying the aio_wake, aio_wait.  Since it is mutually
exclusive with a callback completion, we can avoid calling it at all when
a callback in present, and focus on keeping it simple.

Avoid use-after-free by making sure the last decrement happens under
the lock in the aio_wake/wait case.

Signed-off-by: Sage Weil <sage@redhat.com>

If we are a syncrhonous read, we don't need this: we don't aio_wait for
sync reads.  If we are an aio_read, we are in the aio_running count anyway,
and there is also no purpose for this counter.

I'm a bit unsure about the NVME use of this counter; I switched it to use
num_running (pretty sure we aren't mixing reads and writes on a single
IOContext) *but* it might make more sense to switch to a private counter.

Signed-off-by: Sage Weil <sage@redhat.com>

rbd: stop indefinite thread waiting in krbd udev handling

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

This works out of the box with a vstart environment and

 RGW=1 ../src/vstart.sh -n -l
 PATH=bin:$PATH ../qa/workunits/rgw/run-s3tests.sh

Signed-off-by: Sage Weil <sage@redhat.com>

…lation

Script that generates corpus objects.

Signed-off-by: Sage Weil <sage@redhat.com>

list_plain_entries() was using encode_obj_versioned_data_key() to set
its end_key, which gives a prefix of BI_BUCKET_OBJ_INSTANCE_INDEX[=2]

that range between start_key and end_key would not only span the
BI_BUCKET_OBJS_INDEX[=0] prefixes, but BI_BUCKET_LOG_INDEX[=1] prefixes
as well. this can result in list_plain_entries() trying and failing to
decode a rgw_bi_log_entry as a rgw_bucket_dir_entry

Fixes: http://tracker.ceph.com/issues/19876

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Signed-off-by: Sage Weil <sage@redhat.com>

cls/rgw: list_plain_entries() stops before bi_log entries
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

osd-dup.sh was losing OSDs to EMFILE, mostly on arm64 (not clear why).

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Signed-off-by: Sage Weil <sage@redhat.com>

Wow, this has been broken since v0.38, but apparently
the message never made it into the object corpus so
we never noticed!

In reality the bug is harmless: decode_message() will
set_header which clobbers whatever version the default
ctor fills in, so this only affects ceph-dencoder's
test.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: zipkin tracing

os/bluestore/BlueFS: add bluefs_sync_write option

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

ceph: perfcounter priorities and daemonperf updates to use them

common/Finisher: fix uninitialized variable warning
common/Finisher: fix uninitialized variable warning

Reviewed-by: Kefu Chai <kchai@redhat.com>

mon: fix prime_pg_temp overrun
common/Finisher: fix uninitialized variable warning

Reviewed-by: Kefu Chai <kchai@redhat.com>

Also catches corner-case found by Zheng where an unjournaled directory will
cause export pinning to fail because it cannot be made a subtree until its
parent is stable.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

If the dirfrag is pinned to the current mds, it needs to be made into an aux
subtree.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Signed-off-by: Zhou Zhengping <johnzzpcrystal@gmail.com>

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

crush: update documentation for negative choose step

Reviewed-by: Loic Dachary <ldachary@redhat.com>

it's simpler and easier to manage the life cycle of them this way.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Fixes: http://tracker.ceph.com/issues/19813
Signed-off-by: Kefu Chai <kchai@redhat.com>

Introduced by aebc1ca. Found by Zheng.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

…ilable

osd: sched_scrub() lock pg only if all scrubbing conditions are fulfilled

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Pan Liu <liupan1111@gmail.com>

mon: check is_shutdown() in timer callbacks

Reviewed-by: Sage Weil <sage@redhat.com>

it was restructured by 3cc4827

Signed-off-by: Wei Jin <wjin.cn@gmail.com>

os/bluestore: fix use after free race with aio_wait

Reviewed-by: Haomai Wang <haomai@xsky.com>
Reviewed-by: Jianpeng Ma <jianpeng@intel.com>

cherrypy.engine.stop() doesn't actually completely kill cherrypy,
which means cherrypy.engine.block() never returns, so this module's
thread never completes cleanly.

Signed-off-by: Tim Serong <tserong@suse.com>

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>