Question about the correct usage of AddMember and Value #412

whitegfx · 2015-08-25T13:32:57Z

Hi to all I have a question regarding the correct usage of AddMember and Value.
I read doc many times, how to create Value and add member with StringRef but still random crashes. (1, 2) they crash, (3,4) I hope that they are ok.

So please correct me if I use the methods in the wrong way. Or link with more examples with different literal and constructs types are welcome.
RAPIDJSON_HAS_STDSTRING is set to default (0).

Error that i get many times but random, differ on used addMember or Value function.

(rapidjson::GenericValuerapidjson::UTF8<char, rapidjson::MemoryPoolAllocatorrapidjson::CrtAllocator >::AddMember(rapidjson::GenericStringRef, rapidjson::GenericValuerapidjson::UTF8<char, rapidjson::MemoryPoolAllocatorrapidjson::CrtAllocator >&&, rapidjson::MemoryPoolAllocatorrapidjson::CrtAllocator&)+66)

/// defined in app scope
static const char* LONG_LIVING_KEY = "LONG_LIVING_KEY";
static const char* LONG_LIVING_VALUE = "LONG_LIVING_VALUE";

// utility to convert json to string representation
template<>
std::string AHXLocalData::serialize< rapidjson::Document >(const rapidjson::Document &data){
    rapidjson::StringBuffer strbuf;
    rapidjson::Writer <rapidjson::StringBuffer> writer(strbuf);
    data.Accept(writer);
    return strbuf.GetString();
}

// crash
// using static const char* as key and value with StringRef
// inside class method
rapidjson::Document doc;
doc.SetObject();
rapidjson::Document::AllocatorType& allocator = doc.GetAllocator();
doc.AddMember(rapidjson::StringRef(LONG_LIVING_KEY), rapidjson::Value(rapidjson::StringRef(LONG_LIVING_VALUE)), allocator);
// will crash sometimes on rapidjson::Value( and addMember(
logEvent(serialize(doc).c_str());

// crash
// using static const char* as key and value, StringRef only on key
// inside class method
rapidjson::Document doc;
doc.SetObject();
rapidjson::Document::AllocatorType& allocator = doc.GetAllocator();
doc.AddMember(rapidjson::StringRef(LONG_LIVING_KEY), rapidjson::Value(LONG_LIVING_VALUE), allocator);
// will crash sometimes on rapidjson::Value( and addMember(
logEvent(serialize(doc).c_str());

// not crash
// using static const char* as key , value created from std::string
// inside class method
rapidjson::Document doc;
doc.SetObject();

std::string levelName = getTheLevelName(); // return std::string
rapidjson::Value rjv;
rjv.SetString(levelName.c_str(), allocator)
doc.AddMember(rapidjson::StringRef(LONG_LIVING_KEY), rjv, allocator);
logEvent(serialize(doc).c_str());

// not crash
// using compiled literal key and value
// inside class method
rapidjson::Document doc;
doc.SetObject();

rapidjson::Value rjv;
doc.AddMember("from", "dialog", allocator);
logEvent(serialize(doc).c_str());

Thank you for your time.

The text was updated successfully, but these errors were encountered:

whitegfx · 2015-09-02T20:15:25Z

Small update, I tested small code on iOS, Android and Kindle Fire HDX with this result.
A)

rapidjson::Document doc;
doc.SetObject();
rapidjson::Document::AllocatorType& allocator = doc.GetAllocator();
doc.AddMember("ccc","xxx",allocator);

On iOS iPhone or iPad, Android no problem. On Kindle crash every time in addMember function.
When I comment //doc.SetObject(); code works.

B)

/// defined in app scope
static const char* LONG_LIVING_KEY = "LONG_LIVING_KEY";
static const char* LONG_LIVING_VALUE = "LONG_LIVING_VALUE";
//...
rapidjson::Document doc;
doc.SetObject();
rapidjson::Document::AllocatorType& allocator = doc.GetAllocator();
doc.AddMember(rapidjson::StringRef(LONG_LIVING_KEY), rapidjson::StringRef(LONG_LIVING_VALUE),allocator);

On iOS iPhone or iPad, no problem. On Android I have random crash. On Kindle crash every time.

C)

rapidjson::CrtAllocator crtAllocator;
rapidjson::GenericDocument<rapidjson::UTF8<>, rapidjson::CrtAllocator> doc(&crtAllocator);
doc.SetObject();
rapidjson::CrtAllocator& allocator = doc.GetAllocator();
doc.AddMember(
              rapidjson::StringRef(LOG_EVENT_NameCompletedRegistration),
              rapidjson::StringRef(STORE_NAME),
              allocator);

After reading all rapidjson issues I found (C) code. And works on all tested platforms.
Whats wrong? Do I need to setup something in rapidjson to avoid problems with allocator for android and kindle?

Thx

miloyip · 2015-09-03T04:04:38Z

Thank you for your report.

I think the use of StringRef is correct is your code. The problem seems related to memory allocation in some platforms.

There has been some similar reports in the past but no diagnosis can be found. Using CrtAllocator seem have fixed in those reports as well.

Since you found that in some cases it always crash, that may be a very good opportunity to dig out the cause exactly. I hope you may help diagnoses how it is crashed. It may be related to address alignment or similar issues. I hope we can work together to resolve this.

whitegfx · 2015-09-03T10:00:13Z

Hi here is the test for different memory align no change for now, still crash on Kindle. I tried current master branch and 1.0.2 from cocos2d-x. For both rapidjson version I have same result.

Here are dumps with master branch:
https://gist.github.com/whitegfx/b68100ce366f09ba7c46

Maybe this will help. I saw that someone report similar issue on arm only in release mode.
So I try to test it. When project is compiled with:

-fexceptions -frtti -std=c++11 -fsigned-char
APP_OPTIM := debug

There is no crash.

Crash occurs only with these flags.

-frtti -std=c++11 -fsigned-char
APP_OPTIM := release

Used in test: mac OS X 10.10.5, ndk toolchain: 4.9, android target 22, min sdk 19.
If you need more information or perform some test (please no ;8 rapidjson unitest ), let me know.

miloyip · 2015-09-04T03:12:10Z

The dump shows that rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator>::Malloc(unsigned int)+7 generated the crash.

Since it only happens in release mode, it may be hard to gdb or something. Please try printing the intermediate variables (via some kind of printf() & fflush(stdout) to console or file) in that Malloc() and AddChunk().

In class MemoryPoolAllocator,

    void* Malloc(size_t size) {
        printf("Malloc(%zu)\n", size); fflush(stdout);

        if (!size)
            return NULL;

        size = RAPIDJSON_ALIGN(size);
        printf("aligned size = %zu\n", size); fflush(stdout);

        if (chunkHead_ == 0 || chunkHead_->size + size > chunkHead_->capacity)
            AddChunk(chunk_capacity_ > size ? chunk_capacity_ : size);

        void *buffer = reinterpret_cast<char *>(chunkHead_) + RAPIDJSON_ALIGN(sizeof(ChunkHeader)) + chunkHead_->size;
        chunkHead_->size += size;

        printf("buffer = %p", buffer); fflush(stdout);

        return buffer;
    }

    // ...

    void AddChunk(size_t capacity) {
        printf("AddChunk(%zu)\n", capacity); fflush(stdout);

        if (!baseAllocator_) {
            printf("Going to new BaseAllocator()"); fflush(stdout);
            ownBaseAllocator_ = baseAllocator_ = RAPIDJSON_NEW(BaseAllocator());
            printf("ownBaseAllocator = %p\n", ownBaseAllocator); fflush(stdout);
        }
        size_t s = RAPIDJSON_ALIGN(sizeof(ChunkHeader)) + capacity;
        printf("s = %zu\n", s); fflush(stdout);
        ChunkHeader* chunk = reinterpret_cast<ChunkHeader*>(baseAllocator_->Malloc(s));
        printf("chunk = %p", chunk); fflush(stdout);
        chunk->capacity = capacity;
        chunk->size = 0;
        chunk->next = chunkHead_;
        chunkHead_ =  chunk;
    }

Thank you.

whitegfx · 2015-09-04T08:31:14Z

Hi, I replace your log with LOGD. In the dump all is ok until this code is accesed. App crash when chunkHead_ is accessed for some reason. Im unable to log something about chunkHead_ like if (chunkHead_ == 0 or !=0) always crash. But as you can see other work with rapidjson before with malloc or add chunk works fine.

rapidjson::Document doc;
doc.SetObject();
rapidjson::Document::AllocatorType& allocator = doc.GetAllocator();
doc.AddMember("ccc","xxx",allocator);

void* Malloc(size_t size) {
    LOGD("Malloc(%zu)", size);

    if (!size)
        return NULL;

    size = RAPIDJSON_ALIGN(size);
    LOGD("aligned size = %zu", size);

    if (chunkHead_ == 0 || chunkHead_->size + size > chunkHead_->capacity)
        AddChunk(chunk_capacity_ > size ? chunk_capacity_ : size);

    void *buffer = reinterpret_cast<char *>(chunkHead_) + RAPIDJSON_ALIGN(sizeof(ChunkHeader)) + chunkHead_->size;
    chunkHead_->size += size;

    LOGD("buffer = %p", buffer);

    return buffer;
}

void AddChunk(size_t capacity) {
    LOGD("AddChunk(%zu)", capacity);

    if (!baseAllocator_) {
        LOGD("Going to new BaseAllocator()");
        ownBaseAllocator_ = baseAllocator_ = RAPIDJSON_NEW(BaseAllocator());
        LOGD("ownBaseAllocator = %p", ownBaseAllocator_);
    }
    size_t s = RAPIDJSON_ALIGN(sizeof(ChunkHeader)) + capacity;
    LOGD("s = %zu", s);
    ChunkHeader* chunk = reinterpret_cast<ChunkHeader*>(baseAllocator_->Malloc(s));
    LOGD("chunk = %p", chunk);
    chunk->capacity = capacity;
    chunk->size = 0;
    chunk->next = chunkHead_;
    chunkHead_ =  chunk;
}

Here is the dump.
https://gist.github.com/whitegfx/8289534168eebddb8dbb

I tried to duplicate code like this, and after that crash is very random.

{
rapidjson::Document doc;
doc.SetObject();
rapidjson::Document::AllocatorType& allocator = doc.GetAllocator();
doc.AddMember("ccc","xxx",allocator);
}
rapidjson::Document doc;
doc.SetObject();
rapidjson::Document::AllocatorType& allocator = doc.GetAllocator();
doc.AddMember("ccc","xxx",allocator);

When there is no crash code continue with this log (09-04 10:22:34.222).
https://gist.github.com/whitegfx/a256ab46c9585c0caf46

To be sure I clean function to have only rj code, function is called every 5 sec from timer.
Dump for this case is: https://gist.github.com/whitegfx/4cb762141a8f33cc3802

void MainDialog::checkTask(float dt){
    rapidjson::Document doc;
    doc.SetObject();
    rapidjson::Document::AllocatorType& allocator = doc.GetAllocator();
    doc.AddMember("ccc","xxx",allocator);
    return;
}

Another test, MainDialog::checkTask is called after 20sec. Time to process all network task before fce is called.
Dump: https://gist.github.com/whitegfx/7fe4b93d63c3c1a92447

miloyip · 2015-09-04T13:26:02Z

From

Im unable to log something about chunkHead_ like if (chunkHead_ == 0 or !=0)

I suspect that this is NULL because chunkHead_ is the first member of MemoryPoolAllocator thus having offset of zero and the dump

Fatal signal 11 (SIGSEGV) at 0x00000000 (code=1)

may relate to reading at the null pointer. You may check (this == 0) at the beginning of the functions.

I think CrtAllocator does not crash because it does not have member variables nor virtual functions.

Hum. Then the root of problem should be at the constructor of GenericDocument which fails to new a allocator. You may log that part after verifying the above null pointer issue.

whitegfx · 2015-09-04T13:55:38Z

You are right, is null. That the reason why always crash when I acces chunkHead_. Let me know what we need to debug in document.h

void* Malloc(size_t size) {
LOGD("Malloc this == 0 %d", (this == 0));
LOGD("Malloc(%zu)", size);

dump:
https://gist.github.com/whitegfx/f91252ef2d2f3abea755

miloyip · 2015-09-04T14:33:00Z

The MemberPoolAllocator instance is new in GenericDocument's constructors:

    explicit GenericDocument(Type type, Allocator* allocator = 0, size_t stackCapacity = kDefaultStackCapacity, StackAllocator* stackAllocator = 0) :
        GenericValue<Encoding, Allocator>(type),  allocator_(allocator), ownAllocator_(0), stack_(stackAllocator, stackCapacity), parseResult_()
    {
        if (!allocator_)
            ownAllocator_ = allocator_ = RAPIDJSON_NEW(Allocator());
    }

    GenericDocument(Allocator* allocator = 0, size_t stackCapacity = kDefaultStackCapacity, StackAllocator* stackAllocator = 0) : 
        allocator_(allocator), ownAllocator_(0), stack_(stackAllocator, stackCapacity), parseResult_()
    {
        if (!allocator_)
            ownAllocator_ = allocator_ = RAPIDJSON_NEW(Allocator());
    }

I am quite sure that the new returns null. But need to confirm this. Or whether the constructor of MemoryPoolAllocator is called.

However, it seems quite strange whatever these happens...

mloskot · 2015-09-04T19:07:09Z

Perhaps on some mobile platforms new is non-throwing by default.
Simple check should make such situation leak:

if (!allocator_)
{
    ownAllocator_ = allocator_ = RAPIDJSON_NEW(Allocator());
    if (!ownAllocator_ || !allocator_) throw __LINE__;
}

whitegfx · 2015-09-06T12:10:40Z

@mloskot exceptions are disabled and crash happen only in release mode not in debug. When excp enabled in release mode there are no debug information due release mode. Only what we get from backtrace.

@miloyip I loged both allocators and defautl ctor is called, but after that, allocator Malloc this == 0 is true, is it possible?

MemoryPoolAllocator(size_t chunkSize = kDefaultChunkCapacity, BaseAllocator* baseAllocator = 0) : 
    chunkHead_(0), chunk_capacity_(chunkSize), userBuffer_(0), baseAllocator_(baseAllocator), ownBaseAllocator_(0)
{
    LOGD("default MemoryPoolAllocator called %i", 1);
}

void* Malloc(size_t size) {
    LOGD("Malloc this == 0 %d", (this == 0));
    LOGD("Malloc(%zu)", size);

    if (!size)
        return NULL;

    size = RAPIDJSON_ALIGN(size);
    LOGD("aligned size = %zu", size);

    if (chunkHead_ == 0 || chunkHead_->size + size > chunkHead_->capacity)
        AddChunk(chunk_capacity_ > size ? chunk_capacity_ : size);

    void *buffer = reinterpret_cast<char *>(chunkHead_) + RAPIDJSON_ALIGN(sizeof(ChunkHeader)) + chunkHead_->size;
    chunkHead_->size += size;

    LOGD("buffer = %p", buffer);

    return buffer;
}

09-06 12:50:22.374  12154-12178/rjtest D/allocators.h﹕ default MemoryPoolAllocator called 1
09-06 12:50:22.374  12154-12178/rjtest D/allocators.h﹕ Malloc this == 0 1
09-06 12:50:22.374  12154-12178/rjtest D/allocators.h﹕ Malloc(768)
09-06 12:50:22.384  12154-12178/rjtest D/allocators.h﹕ aligned size = 768
09-06 12:50:22.384  12154-12178/rjtest A/libc﹕ Fatal signal 11 (SIGSEGV) at 0x00000000 (code=1), thread 12178 (Thread-5300)

Some log before crash works fine.

9-06 12:50:21.414  12154-12178/rjtest D/allocators.h﹕ Malloc(41)
09-06 12:50:21.414  12154-12178/rjtest D/allocators.h﹕ aligned size = 44
09-06 12:50:21.414  12154-12178/rjtest D/allocators.h﹕ buffer = 0x7059d044
09-06 12:50:21.414  12154-12178/rjtest D/allocators.h﹕ Malloc this == 0 0
09-06 12:50:21.414  12154-12178/rjtest D/allocators.h﹕ Malloc(96)
09-06 12:50:21.414  12154-12178/rjtest D/allocators.h﹕ aligned size = 96
09-06 12:50:21.414  12154-12178/rjtest D/allocators.h﹕ buffer = 0x7059d070
09-06 12:50:21.414  12154-12178/rjtest D/allocators.h﹕ default MemoryPoolAllocator called 1
09-06 12:50:21.414  12154-12178/rjtest D/allocators.h﹕ Malloc this == 0 0
09-06 12:50:21.414  12154-12178/rjtest D/allocators.h﹕ Malloc(22)
09-06 12:50:21.414  12154-12178/rjtest D/allocators.h﹕ aligned size = 24
09-06 12:50:21.414  12154-12178/rjtest D/allocators.h﹕ AddChunk this == 0 0
09-06 12:50:21.414  12154-12178/rjtest D/allocators.h﹕ AddChunk(65536)

Very strange is when I add two LOGD functions all works, without them code always crash.

if (!allocator_){
    ownAllocator_ = allocator_ = RAPIDJSON_NEW(Allocator());
    LOGD("explicit GenericDocument allocator_ == 0 %i", (allocator_ == 0));
    LOGD("explicit GenericDocument ownAllocator_ == 0 %i", (ownAllocator_ == 0));
}
if (!allocator_){
    ownAllocator_ = allocator_ = RAPIDJSON_NEW(Allocator());
    LOGD("GenericDocument allocator_ == 0 %i", (allocator_ == 0));
    LOGD("GenericDocument ownAllocator_ == 0 %i", (ownAllocator_ == 0));
}

Here is the log with two LOGD functions.

09-06 13:08:36.584  19938-19953/rjtest D/allocators.h﹕ default MemoryPoolAllocator called 1
09-06 13:08:36.584  19938-19953/rjtest D/allocators.h﹕ GenericDocument allocator_ == 0 0
09-06 13:08:36.584  19938-19953/rjtest D/allocators.h﹕ GenericDocument ownAllocator_ == 0 0
09-06 13:08:36.584  19938-19953/rjtest D/allocators.h﹕ Malloc this == 0 0
09-06 13:08:36.584  19938-19953/rjtest D/allocators.h﹕ Malloc(768)
09-06 13:08:36.584  19938-19953/rjtest D/allocators.h﹕ aligned size = 768
09-06 13:08:36.584  19938-19953/rjtest D/allocators.h﹕ AddChunk this == 0 0
09-06 13:08:36.584  19938-19953/rjtest D/allocators.h﹕ AddChunk(65536)
09-06 13:08:36.584  19938-19953/rjtest D/allocators.h﹕ Going to new BaseAllocator()
09-06 13:08:36.584  19938-19953/rjtest D/allocators.h﹕ ownBaseAllocator = 0x61be8d38
09-06 13:08:36.584  19938-19953/rjtest D/allocators.h﹕ s = 65548
09-06 13:08:36.584  19938-19953/rjtest D/allocators.h﹕ chunk = 0x6c978cd8
09-06 13:08:36.584  19938-19953/rjtest D/allocators.h﹕ buffer = 0x6c978ce4
09-06 13:08:36.584  19938-19953/rjtest D/allocators.h﹕ default MemoryPoolAllocator called 1
09-06 13:08:36.584  19938-19953/rjtest D/allocators.h﹕ GenericDocument allocator_ == 0 0
09-06 13:08:36.584  19938-19953/rjtest D/allocators.h﹕ GenericDocument ownAllocator_ == 0 0
09-06 13:08:36.984  19938-19953/rjtest D/allocators.h﹕ default MemoryPoolAllocator called 1
09-06 13:08:36.984  19938-19953/rjtest D/allocators.h﹕ GenericDocument allocator_ == 0 0
09-06 13:08:36.984  19938-19953/rjtest D/allocators.h﹕ GenericDocument ownAllocator_ == 0 0
09-06 13:08:36.984  19938-19953/rjtest D/allocators.h﹕ Malloc this == 0 0
09-06 13:08:36.984  19938-19953/rjtest D/allocators.h﹕ Malloc(13)
09-06 13:08:36.984  19938-19953/rjtest D/allocators.h﹕ aligned size = 16
09-06 13:08:36.984  19938-19953/rjtest D/allocators.h﹕ AddChunk this == 0 0
09-06 13:08:36.984  19938-19953/rjtest D/allocators.h﹕ AddChunk(65536)
09-06 13:08:36.984  19938-19953/rjtest D/allocators.h﹕ Going to new BaseAllocator()
09-06 13:08:36.984  19938-19953/rjtest D/allocators.h﹕ ownBaseAllocator = 0x69a54f70
09-06 13:08:36.984  19938-19953/rjtest D/allocators.h﹕ s = 65548
09-06 13:08:36.984  19938-19953/rjtest D/allocators.h﹕ chunk = 0x69c4b3d0
09-06 13:08:36.984  19938-19953/rjtest D/allocators.h﹕ buffer = 0x69c4b3dc
09-06 13:08:36.984  19938-19953/rjtest D/allocators.h﹕ Malloc this == 0 0
09-06 13:08:36.984  19938-19953/rjtest D/allocators.h﹕ Malloc(31)
09-06 13:08:36.984  19938-19953/rjtest D/allocators.h﹕ aligned size = 32
09-06 13:08:36.984  19938-19953/rjtest D/allocators.h﹕ buffer = 0x69c4b3ec
09-06 13:08:36.984  19938-19953/rjtest D/allocators.h﹕ Malloc this == 0 0
09-06 13:08:36.984  19938-19953/rjtest D/allocators.h﹕ Malloc(41)
09-06 13:08:36.984  19938-19953/rjtest D/allocators.h﹕ aligned size = 44
09-06 13:08:36.984  19938-19953/rjtest D/allocators.h﹕ buffer = 0x69c4b40c
09-06 13:08:36.984  19938-19953/rjtest D/allocators.h﹕ Malloc this == 0 0
09-06 13:08:36.984  19938-19953/rjtest D/allocators.h﹕ Malloc(96)
09-06 13:08:36.984  19938-19953/rjtest D/allocators.h﹕ aligned size = 96
09-06 13:08:36.984  19938-19953/rjtest D/allocators.h﹕ buffer = 0x69c4b438

Commented LOGD for allocator_ in both constructors, still works. Also when commented second and uncommented first LOGD still works.

if (!allocator_){
    ownAllocator_ = allocator_ = RAPIDJSON_NEW(Allocator());
    //LOGD("explicit GenericDocument allocator_ == 0 %i", (allocator_ == 0));
    LOGD("explicit GenericDocument ownAllocator_ == 0 %i", (ownAllocator_ == 0));
}
if (!allocator_){
    ownAllocator_ = allocator_ = RAPIDJSON_NEW(Allocator());
    //LOGD("GenericDocument allocator_ == 0 %i", (allocator_ == 0));
    LOGD("GenericDocument ownAllocator_ == 0 %i", (ownAllocator_ == 0));
}

09-06 13:23:01.484  27562-27583/rjtest D/allocators.h﹕ default MemoryPoolAllocator called 1
09-06 13:23:01.484  27562-27583/rjtest D/allocators.h﹕ GenericDocument ownAllocator_ == 0 0
09-06 13:23:01.484  27562-27583/rjtest D/allocators.h﹕ Malloc this == 0 0
09-06 13:23:01.484  27562-27583/rjtest D/allocators.h﹕ Malloc(768)
09-06 13:23:01.484  27562-27583/rjtest D/allocators.h﹕ aligned size = 768
09-06 13:23:01.484  27562-27583/rjtest D/allocators.h﹕ AddChunk this == 0 0
09-06 13:23:01.484  27562-27583/rjtest D/allocators.h﹕ AddChunk(65536)
09-06 13:23:01.484  27562-27583/rjtest D/allocators.h﹕ Going to new BaseAllocator()
09-06 13:23:01.484  27562-27583/rjtest D/allocators.h﹕ ownBaseAllocator = 0x6c2748e0
09-06 13:23:01.484  27562-27583/rjtest D/allocators.h﹕ s = 65548
09-06 13:23:01.484  27562-27583/rjtest D/allocators.h﹕ chunk = 0x6c2852a0
09-06 13:23:01.484  27562-27583/rjtest D/allocators.h﹕ buffer = 0x6c2852ac
09-06 13:23:01.484  27562-27583/rjtest D/allocators.h﹕ default MemoryPoolAllocator called 1
09-06 13:23:01.484  27562-27583/rjtest D/allocators.h﹕ GenericDocument ownAllocator_ == 0 0
09-06 13:23:08.494  27562-27583/rjtest D/allocators.h﹕ default MemoryPoolAllocator called 1
09-06 13:23:08.494  27562-27583/rjtest D/allocators.h﹕ GenericDocument ownAllocator_ == 0 0
09-06 13:23:08.494  27562-27583/rjtest D/allocators.h﹕ Malloc this == 0 0
09-06 13:23:08.494  27562-27583/rjtest D/allocators.h﹕ Malloc(768)
09-06 13:23:08.494  27562-27583/rjtest D/allocators.h﹕ aligned size = 768
09-06 13:23:08.494  27562-27583/rjtest D/allocators.h﹕ AddChunk this == 0 0
09-06 13:23:08.494  27562-27583/rjtest D/allocators.h﹕ AddChunk(65536)
09-06 13:23:08.494  27562-27583/rjtest D/allocators.h﹕ Going to new BaseAllocator()
09-06 13:23:08.494  27562-27583/rjtest D/allocators.h﹕ ownBaseAllocator = 0x6c2748e0
09-06 13:23:08.494  27562-27583/rjtest D/allocators.h﹕ s = 65548
09-06 13:23:08.494  27562-27583/rjtest D/allocators.h﹕ chunk = 0x6c2852a0
09-06 13:23:08.494  27562-27583/rjtest D/allocators.h﹕ buffer = 0x6c2852ac
09-06 13:23:08.504  27562-27583/rjtest D/allocators.h﹕ default MemoryPoolAllocator called 1
09-06 13:23:08.504  27562-27583/rjtest D/allocators.h﹕ GenericDocument ownAllocator_ == 0 0
09-06 13:23:15.504  27562-27583/rjtest D/allocators.h﹕ default MemoryPoolAllocator called 1
09-06 13:23:15.504  27562-27583/rjtest D/allocators.h﹕ GenericDocument ownAllocator_ == 0 0
09-06 13:23:15.504  27562-27583/rjtest D/allocators.h﹕ Malloc this == 0 0
09-06 13:23:15.504  27562-27583/rjtest D/allocators.h﹕ Malloc(768)
09-06 13:23:15.504  27562-27583/rjtest D/allocators.h﹕ aligned size = 768
09-06 13:23:15.504  27562-27583/rjtest D/allocators.h﹕ AddChunk this == 0 0
09-06 13:23:15.504  27562-27583/rjtest D/allocators.h﹕ AddChunk(65536)
09-06 13:23:15.504  27562-27583/rjtest D/allocators.h﹕ Going to new BaseAllocator()
09-06 13:23:15.504  27562-27583/rjtest D/allocators.h﹕ ownBaseAllocator = 0x6badf4c0
09-06 13:23:15.514  27562-27583/rjtest D/allocators.h﹕ s = 65548
09-06 13:23:15.514  27562-27583/rjtest D/allocators.h﹕ chunk = 0x6c2852a0

Both commented added another LOGD before call new, and again Malloc this == 0 is true for allocator after ctor was called. Any tip why allocator is null after his ctor is called?

if (!allocator_){
    LOGD("explicit GenericDocument call %i", 1);
    ownAllocator_ = allocator_ = RAPIDJSON_NEW(Allocator());
    //LOGD("explicit GenericDocument allocator_ == 0 %i", (allocator_ == 0));
    //LOGD("explicit GenericDocument ownAllocator_ == 0 %i", (ownAllocator_ == 0));
}
if (!allocator_){
    LOGD("GenericDocument call %i", 1);
    ownAllocator_ = allocator_ = RAPIDJSON_NEW(Allocator());
    //LOGD("GenericDocument allocator_ == 0 %i", (allocator_ == 0));
    //LOGD("GenericDocument ownAllocator_ == 0 %i", (ownAllocator_ == 0));
}

09-06 13:54:21.784  11128-11143/rjtest D/allocators.h﹕ GenericDocument call 1
09-06 13:54:21.784  11128-11143/rjtest D/allocators.h﹕ default MemoryPoolAllocator called 1
09-06 13:54:21.784  11128-11143/rjtest D/allocators.h﹕ Malloc this == 0 1
09-06 13:54:21.784  11128-11143/rjtest D/allocators.h﹕ Malloc(768)
09-06 13:54:21.784  11128-11143/rjtest D/allocators.h﹕ aligned size = 768
09-06 13:54:21.784  11128-11143/rjtest A/libc﹕ Fatal signal 11 (SIGSEGV) at 0x00000000 (code=1), thread 11143 (Thread-5390)
09-06 13:54:21.884      263-263/? I/DEBUG﹕ *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
09-06 13:54:21.884      263-263/? I/DEBUG﹕ Build fingerprint: 'Amazon/thor/thor:4.4.3/KTU84M/13.4.5.5_user_455001820:user/release-keys'
09-06 13:54:21.884      263-263/? I/DEBUG﹕ Revision: '0'
09-06 13:54:21.884      263-263/? I/DEBUG﹕ pid: 11128, tid: 11143, name: Thread-5390  >>> rjtest <<<
09-06 13:54:21.884      263-263/? I/DEBUG﹕ signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 00000000
09-06 13:54:21.984      263-263/? I/DEBUG﹕ r0 00000021  r1 6f2f4a07  r2 6f2f4a07  r3 6f2f4a07
09-06 13:54:21.984      263-263/? I/DEBUG﹕ r4 00000000  r5 60b73e4e  r6 00000300  r7 00000001
09-06 13:54:21.984      263-263/? I/DEBUG﹕ r8 60f46b08  r9 60e48f1c  sl 60e27018  fp 60f46b1c
09-06 13:54:21.984      263-263/? I/DEBUG﹕ ip 60f464dc  sp 60f46950  lr 609662af  pc 609662ae  cpsr 60070030
09-06 13:54:21.984      263-263/? I/DEBUG﹕ d0  0000000000000000  d1  0000000000000000
09-06 13:54:21.984      263-263/? I/DEBUG﹕ d2  61636f6c6c416c6f  d3  6c6c616320726f74
09-06 13:54:21.984      263-263/? I/DEBUG﹕ d4  c012a062be8e6afc  d5  3d9e75b93f41d644
09-06 13:54:21.984      263-263/? I/DEBUG﹕ d6  400000003f41d644  d7  312777863331bb4c
09-06 13:54:21.984      263-263/? I/DEBUG﹕ d8  0000000000000000  d9  0000000000000000
09-06 13:54:21.984      263-263/? I/DEBUG﹕ d10 0000000000000000  d11 0000000000000000
09-06 13:54:21.984      263-263/? I/DEBUG﹕ d12 0000000000000000  d13 0000000000000000
09-06 13:54:21.984      263-263/? I/DEBUG﹕ d14 0000000000000000  d15 0000000000000000
09-06 13:54:21.984      263-263/? I/DEBUG﹕ d16 7420636f6c6c614d  d17 30203d3d20736968
09-06 13:54:21.984      263-263/? I/DEBUG﹕ d18 7cb5b65493bc889b  d19 7247792cd98c1c51
09-06 13:54:21.984      263-263/? I/DEBUG﹕ d20 9294f340583e4f93  d21 9e0d73d4e90aaf56
09-06 13:54:21.984      263-263/? I/DEBUG﹕ d22 6d4da4161809a66b  d23 889dc4a4214959c5
09-06 13:54:21.984      263-263/? I/DEBUG﹕ d24 bf29f834d9d1e8bc  d25 3fb89d10b50efa40
09-06 13:54:21.984      263-263/? I/DEBUG﹕ d26 bfd3874101d65ac1  d27 bf9e87375408c742
09-06 13:54:21.984      263-263/? I/DEBUG﹕ d28 3f82ee9bfb8fd367  d29 3ec6cd878c3b46a7
09-06 13:54:21.984      263-263/? I/DEBUG﹕ d30 bf320fb199b7c052  d31 3ef99342e0ee5069
09-06 13:54:21.984      263-263/? I/DEBUG﹕ scr 8000001a
09-06 13:54:21.984      263-263/? I/DEBUG﹕ backtrace:
09-06 13:54:21.984      263-263/? I/DEBUG﹕ #00  pc 001332ae  /data/app-lib/rjtest-1/libcocos2dcpp.so (rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator>::Malloc(unsigned int)+65)
09-06 13:54:21.984      263-263/? I/DEBUG﹕ #01  pc 00146a73  /data/app-lib/rjtest-1/libcocos2dcpp.so (rapidjson::GenericValue<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator> >::AddMember(rapidjson::GenericStringRef<char>, rapidjson::GenericValue<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator> >&, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator>&)+66)
09-06 13:54:21.984      263-263/? I/DEBUG﹕ #02  pc 00146df1  /data/app-lib/rjtest-1/libcocos2dcpp.so (MainDialog::checkTask(float)+232)

whitegfx · 2015-09-06T21:41:11Z

I get something more from ndk-stack -sym .... about crash

********** Crash dump: **********
Build fingerprint: 'Amazon/thor/thor:4.4.3/KTU84M/13.4.5.5_user_455001820:user/release-keys'
pid: 6686, tid: 6702, name: Thread-6302  >>> rjtest <<<
signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 00000000
Stack frame #00  pc 00133176  /data/app-lib/rjtest-2/libcocos2dcpp.so (rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator>::Malloc(unsigned int)+65): Routine rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator>::Malloc(unsigned int) at cocos2d/external/json/allocators.h:190

We already know that...

void* Malloc(size_t size) {
    LOGD("Malloc this == 0 %d", (this == 0));
    LOGD("Malloc(%zu)", size);

    if (!size)
        return NULL;

    size = RAPIDJSON_ALIGN(size);
    LOGD("aligned size = %zu", size);

    if (chunkHead_ == 0 || chunkHead_->size + size > chunkHead_->capacity) // 190
        AddChunk(chunk_capacity_ > size ? chunk_capacity_ : size);

    void *buffer = reinterpret_cast<char *>(chunkHead_) + RAPIDJSON_ALIGN(sizeof(ChunkHeader)) + chunkHead_->size;
    chunkHead_->size += size;

    LOGD("buffer = %p", buffer);

    return buffer;
}

This is new...

Stack frame #01  pc 001467ef  /data/app-lib/rjtest-2/libcocos2dcpp.so (MainDialog::checkTask(float)+102): Routine rapidjson::GenericValue<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator> >::AddMember(rapidjson::GenericValue<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator> >&, rapidjson::GenericValue<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator> >&, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator>&) at cocos2d/external/json/**document.h:1001**
Crash dump is completed


GenericValue& AddMember(GenericValue& name, GenericValue& value, Allocator& allocator) {
    RAPIDJSON_ASSERT(IsObject());
    RAPIDJSON_ASSERT(name.IsString());

    Object& o = data_.o;
    if (o.size >= o.capacity) {
        if (o.capacity == 0) {
            o.capacity = kDefaultObjectCapacity;
            o.members = reinterpret_cast<Member*>(allocator.Malloc(o.capacity * sizeof(Member))); // 1001
        }
        else {
            SizeType oldCapacity = o.capacity;
            o.capacity += (oldCapacity + 1) / 2; // grow by factor 1.5
            o.members = reinterpret_cast<Member*>(allocator.Realloc(o.members, oldCapacity * sizeof(Member), o.capacity * sizeof(Member)));
        }
    }
    o.members[o.size].name.RawAssign(name);
    o.members[o.size].value.RawAssign(value);
    o.size++;
    return *this;
}

igorzel · 2015-11-05T16:50:27Z

try to use it in this way:

typedef rapidjson::GenericDocument<rapidjson::UTF8<>,rapidjson::CrtAllocator> DocuType;
typedef rapidjson::GenericValue<rapidjson::UTF8<>,rapidjson::CrtAllocator> ElemType;

DocuType doc;
doc.SetArray();

ElemType element(rapidjson::kObjectType);
element.AddMember("Value", ElemType(1234), doc.GetAllocator());

ElemType array(rapidjson::kArrayType);
array.PushBack(element, doc.GetAllocator());

// and so on...

paresy · 2016-01-18T14:25:34Z

I can reproduce the problem on Windows 10, Visual Studio 2013 and enabling Application Verifier for my exe file

miloyip · 2016-01-19T02:06:12Z

@paresy Can you show a minimal reproducible code?

paresy · 2016-01-19T12:36:20Z

Unfortunately i was wrong. The Heap Check option was using a lot of memory, which resulted in malloc returning null. So my comment was more related to this (#414) and not a rapidjson bug. Sorry.

whitegfx · 2016-01-19T12:59:42Z

Hi I will test UserBuffer too, mentioned in #414 if my problem will gone. My files are over cca 6k - still small ;)

jarsj · 2016-11-20T17:19:35Z

@whitegfx did userbuffer solve this for you ? I guess I am facing the same issue (in cocos2d-x)

whitegfx · 2016-11-20T18:29:26Z

Hi @jarsj , no I upgraded to the latest rapidjson and crash is minimal. But there is some news, more users face this problem in different situations (same as me in release mode), take a look into this discussion for the cause and fix/solution. cocos2d/cocos2d-x#16492

jarsj · 2016-11-22T05:15:49Z

Thanks @whitegfx. I was able to eventually fix mine by using CrtAllocator.

On Sun, Nov 20, 2016 at 11:59 PM, whitegfx notifications@github.com wrote:

Hi @jarsj https://github.com/jarsj , no I upgraded to the latest
rapidjson and crash is minimal. But there is some news, more users face
this problem in different situations (same as me in release mode), take a
look into this discussion for the cause and fix/solution.
cocos2d/cocos2d-x#16492
cocos2d/cocos2d-x#16492

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#412 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABsK60uBl14lIsfs70u8Xdw3Ayqtq98Nks5rAJGMgaJpZM4FxwAM
.

Harsh Jain
Founder & CTO,
Crispy Games

LinkedIn : https://www.linkedin.com/in/jarsj

clolsonus · 2021-06-19T19:41:25Z

I think I am seeing this same issue trying to build rapidjson inside the ardupilot build env for STM32.

When I try to call AddMember on a newly created (empty) Object, the object size and capacity are both 0. The code wants to allocate kDefaultObjectCapacity * sizeof(Member) = 16 * 32 = 512 bytes. However, inside CrtAllocator.Malloc() I print the size requested and get 65548 bytes it is trying to malloc. That call to std::malloc(65548) fails and the function returns null.

I temporarily hacked the code to request kDefaultObjectCapacity * sizeof(Member) + 2 (so 514 bytes), but inside CrtAllocator.Malloc() it is still reporting that I asked for 65548 bytes (didn't change even though I requested a different value.)

I can't figure out why we are calling Malloc(512) and inside CrtAllocator.Malloc() the value is received as 65548. My guess is some sort of confusion/conflict about the size_t datatype? But I can't make the math add up to get from 512 passed to 65548 received? It's almost like there is a memory corruption/overrun bug somewhere and the stack is getting corrupted?

The identical code on my linux desktop pc works fine, but I see it also tries to Malloc this same huge nonesense amount of RAM, the main difference is my PC can happily satisfy the request.

Anyone have any ideas?

clolsonus · 2021-06-19T20:55:53Z

Ok, following up to my own message, I think I finally figured out the problem on small mcpu's.

In allocators.h line #263: static const int kDefaultChunkCapacity = 64 * 1024; //!< Default chunk capacity.

This defines how big the actual allocation chunks size is + some additional bytes. My processor can't satisfy a malloc() call for that size buffer.

I changed kDefaultChunkCapacity to a far lower value of 1024, and now my code seems to be running ok.

I don't know if there's a way to auto set this to a reasonable value with #ifdef magic, or maybe people just have to be warned so they can change to an appropriate value themselves?

Hope this helps anyone who's trying to use rapidjson on a small mcpu.

Thanks,.

herhor67 · 2022-09-15T01:23:14Z

@clolsonus Thank you! That was exactly my issue, while writing for ESP32.
In case anyone stumbles on that,
The kDefaultChunkCapacity you mentioned seems now to be set to RAPIDJSON_ALLOCATOR_DEFAULT_CHUNK_CAPACITY macro, which in turn is set to (64*1024). However, it is guarded by #ifndef, so it is possible to define it before including rapidjson headers.
Which is exactly what I did and it works fine.

miloyip added the bug label Sep 3, 2015

miloyip closed this as completed Feb 2, 2016

matthewlehew mentioned this issue Feb 16, 2016

MemoryPoolAllocator crash #548

Closed

michcleary mentioned this issue Apr 11, 2017

Need to have Value with key value pair assigned at construction without document #927

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Question about the correct usage of AddMember and Value #412

Question about the correct usage of AddMember and Value #412

whitegfx commented Aug 25, 2015

whitegfx commented Sep 2, 2015

miloyip commented Sep 3, 2015

whitegfx commented Sep 3, 2015

miloyip commented Sep 4, 2015

whitegfx commented Sep 4, 2015

miloyip commented Sep 4, 2015

whitegfx commented Sep 4, 2015

miloyip commented Sep 4, 2015

mloskot commented Sep 4, 2015

whitegfx commented Sep 6, 2015

whitegfx commented Sep 6, 2015

igorzel commented Nov 5, 2015

paresy commented Jan 18, 2016

miloyip commented Jan 19, 2016

paresy commented Jan 19, 2016

whitegfx commented Jan 19, 2016

jarsj commented Nov 20, 2016

whitegfx commented Nov 20, 2016

jarsj commented Nov 22, 2016

clolsonus commented Jun 19, 2021 •

edited

clolsonus commented Jun 19, 2021 •

edited

herhor67 commented Sep 15, 2022 •

edited

Question about the correct usage of AddMember and Value #412

Question about the correct usage of AddMember and Value #412

Comments

whitegfx commented Aug 25, 2015

whitegfx commented Sep 2, 2015

miloyip commented Sep 3, 2015

whitegfx commented Sep 3, 2015

miloyip commented Sep 4, 2015

whitegfx commented Sep 4, 2015

miloyip commented Sep 4, 2015

whitegfx commented Sep 4, 2015

miloyip commented Sep 4, 2015

mloskot commented Sep 4, 2015

whitegfx commented Sep 6, 2015

whitegfx commented Sep 6, 2015

igorzel commented Nov 5, 2015

paresy commented Jan 18, 2016

miloyip commented Jan 19, 2016

paresy commented Jan 19, 2016

whitegfx commented Jan 19, 2016

jarsj commented Nov 20, 2016

whitegfx commented Nov 20, 2016

jarsj commented Nov 22, 2016

clolsonus commented Jun 19, 2021 • edited

clolsonus commented Jun 19, 2021 • edited

herhor67 commented Sep 15, 2022 • edited

clolsonus commented Jun 19, 2021 •

edited

clolsonus commented Jun 19, 2021 •

edited

herhor67 commented Sep 15, 2022 •

edited