SIGSEGV faults seen when child process exits

[maheshke]

Confirm by changing [ ] to [x] below to ensure that it's a bug:

• I've gone through Developer Guide and API reference
• I've searched for previous similar issues and didn't find any solution

Describe the bug
Application uses fork() to spawn child processes to execute jobs. And when the child process exits after the job completion, it is triggering AWS SDK resource cleanup which intermittently causes SIGSEGV faults.

Below are the two SIGSEGV faults seen so far:

1. Logging sub-system

Thread 5.1 "ebs_direct_poc" received signal SIGSEGV, Segmentation fault.
0x00007ffff74ac4c0 in __pthread_timedjoin_ex () from /lib64/libpthread.so.0
(gdb) bt
#0  0x00007ffff74ac4c0 in __pthread_timedjoin_ex () from /lib64/libpthread.so.0
#1  0x00007ffff71d0e27 in std::thread::join() () from /lib64/libstdc++.so.6
#2  0x00007ffff779c4c7 in Aws::Utils::Logging::DefaultLogSystem::~DefaultLogSystem() () from /usr/local/lib64/libaws-cpp-sdk-core.so
#3  0x00007ffff77d42df in std::shared_ptr<Aws::Utils::Logging::LogSystemInterface>::~shared_ptr() () from /usr/local/lib64/libaws-cpp-sdk-core.so
#4  0x00007ffff67e9037 in __cxa_finalize () from /lib64/libc.so.6
#5  0x00007ffff776a307 in __do_global_dtors_aux () from /usr/local/lib64/libaws-cpp-sdk-core.so
#6  0x00007fffffffdaf0 in ?? ()
#7  0x00007ffff7ddfc96 in _dl_fini () from /lib64/ld-linux-x86-64.so.2
Backtrace stopped: frame did not save the PC

2. CRT Event group shutdown

Thread 6.87 "ebs_direct_poc" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffc9ffb700 (LWP 28581)]
0x00007ffff74ac4c0 in __pthread_timedjoin_ex () from /lib64/libpthread.so.0
(gdb) bt
#0  0x00007ffff74ac4c0 in __pthread_timedjoin_ex () from /lib64/libpthread.so.0
#1  0x00007ffff78abc97 in aws_thread_join () from /usr/local/lib64/libaws-cpp-sdk-core.so
#2  0x00007ffff781b077 in s_destroy () from /usr/local/lib64/libaws-cpp-sdk-core.so
#3  0x00007ffff78168bc in s_aws_event_loop_group_shutdown_sync () from /usr/local/lib64/libaws-cpp-sdk-core.so
#4  0x00007ffff78169a1 in s_event_loop_destroy_async_thread_fn () from /usr/local/lib64/libaws-cpp-sdk-core.so
#5  0x00007ffff78abe47 in thread_fn () from /usr/local/lib64/libaws-cpp-sdk-core.so
#6  0x00007ffff74ab14a in start_thread () from /lib64/libpthread.so.0
#7  0x00007ffff68abdc3 in clone () from /lib64/libc.so.6

SDK version number
v1.9.88

Platform/OS/Hardware/Device
Cent OS 7

To Reproduce (observed behavior)
Snippets from the application which uses EBS client to read blocks from snapshots.

main.cpp:

    Aws::SDKOptions options;
    Aws::InitAPI(options);
    {
        Snapshot *snapshot = new Snapshot(args.snapshot, args.outputFile);

        ProcessExecutor executor = ProcessExecutor(snapshot);
        executor.run();
    }
    Aws::ShutdownAPI(options);

ProcessExecutor.cpp:

ProcessExecutor::run()
{
   childPID = fork());
   if(childPID > 0)
   {
        // wait logic
   }
   else {
        getOperation()->execute();
        exit(0); //child exits here
   }
}

Snapshot.cpp

Snapshot::execute()
{
      // Read blocks from EBS snapshots using Aws::EBS::EBSClient
      // this method completes cleanly
}

Expected behavior
Exit of child processes should not result in SIGSEGV faults.

[jmklix]

Can you include a minimal working example that reproduces the seg faults?

[github-actions]

Greetings! It looks like this issue hasn’t been active in longer than a week. We encourage you to check if this is still an issue in the latest release. Because it has been longer than a week since the last update on this, and in the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please feel free to provide a comment or add an upvote to prevent automatic closure, or if the issue is already closed, please feel free to open a new one.

[maheshke]

@jmklix I have posted minimal code to reproduce this issue at https://github.com/maheshke/aws-cpp-sdk-1779. Let me know if you need more information.

[jmklix]

How often are you getting the segfaults? I'm running the example code you have linked and I'm only getting the following:

Process executor is executing....
Reduced the priority of the process 12701
ListSnapshotBlocks: snap-00ef0e700cc999d7b received 4886 blocks. Next token , total blocks 4886 snapshot size 8GB
ListSnapshotBlocks completed for snap-00ef0e700cc999d7b, total blocks 4886 size: 8GB
Received 0 callbacks
Process 12701 killed by signal: 10.
Child 12701 exited.

[maheshke]

@jmklix, I am getting segfault every time.

aws-cpp-sdk-1779$ date
Fri Nov 12 12:15:44 IST 2021

aws-cpp-sdk-1779$ ./aws-cpp-sdk-1779 --snapshot "snap-025202fa8f7378356" --outputFile temp.out
Process executor is executing....
Reduced the priority of the process 28655
ListSnapshotBlocks: snap-025202fa8f7378356 received 60 blocks. Next token , total blocks 60 snapshot size 1GB
ListSnapshotBlocks completed for snap-025202fa8f7378356, total blocks 60 size: 1GB
Received 0 callbacks
Received 40 callbacks
Received 60 callbacks
Finished writing 60 blocks
Process 28655 killed by signal: 11.
Child 28655 exited.

maheshke@Centos7 aws-cpp-sdk-1779$

In your output, I see signal 10 (SIGUSR1) and zero callbacks which indicates that the child process was killed before completion. Was it killed manually?
If your concern is the time taken to complete then you could either take a small snapshot with very few allocated blocks or remove the get blocks logic from the sample code.