The code from this SDK is too big! easy solution

[jonsmirl]

I am working on an embedded system where I don't have 256GB of RAM to work with, instead I have 128MB. When I link to the AWS SDK it is adding over 30MB to my app. This is because I am using one or two APIs from a dozen different services. If I touch one API in a service the SDK brings in the full API for that service. So my app uses about 20 APIs but support for about 1,000 APIs is getting added to my app.

I did an experiment to reduce the SDK size. I went into the API JSON definition file for S3 and manually deleted the definitions for all of the APIs I was not using. Voila, my app shrank 5MB just from adjusting this one service. Cognito is especially bad, I used one API and it pulls in over 400 APIs.

So, could the API generator be modified to let me specify a list of APIs I need in a file? I could then fill in that file and do cmake -DREGENERATE_CLIENTS=1. And I would magically get an SDK that is about 1/20th the size of my current one.

The linker can not eliminate the unused APIs. I tried putting them all in their own linker segments and it had no effect on removing them. It looks like they are all tied together via virtual functions which stops the linker from tossing them. Deleting the defintions from JSON file is a good solution. I am already using -DBUILD_ONLY and -DMINIMUM_SIZE but need even more shrinkage.

[singku]

Customized model file is supported. But it's a complicated work to do to put different APIs from different services in one file. And use one file instead of multiple files will not make the target much smaller(generated code is the same, just in different files). My suggestion for you is to delete unused apis from the original model files. And strip the target to make it smaller.

On Wed, Nov 8, 2017 at 6:13 PM Jon Smirl ***@***.***> wrote: I am working on an embedded system where I don't have 256GB of RAM to work with, instead I have 128MB. When I link to the AWS SDK it is adding over 30MB to my app. This is because I am using one or two APIs from a dozen different services. If I touch one API in a service the SDK brings in the full API for that service. So my app uses about 20 APIs but support for about 1,000 APIs is getting added to my app. I did an experiment to reduce the SDK size. I went into the API JSON definition file for S3 and manually deleted the definitions for all of the APIs I was not using. Voila, my app shrank 5MB just from adjusting this one service. Cognito is especially bad, I used one API and it pulls in over 400 APIs. So, could the API generator be modified to let me specify a list of APIs I need in a file? I could then fill in that file and do cmake -DREGENERATE_CLIENTS=1. And I would magically get an SDK that is about 1/20th the size of my current one. The linker can not eliminate the unused APIs. I tried putting them all in their own linker segments and it had no effect on removing them. It looks like they are all tied together via virtual functions which stops the linker from tossing them. Deleting the defintions from JSON file is a good solution. I am already using -DBUILD_ONLY and -DMINIMUM_SIZE but need even more shrinkage. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#724>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADTsCEGpGbiJ4nm5cjakdbLfiMM9XhFFks5s0l-2gaJpZM4QXVXZ> .

-- Best Regards! Liang. ------------------ Software Engineer Game Developer

[jonsmirl]

I agree. I am hoping AWS will provide an automated way to strip out the unneeded definitions. I want it automated so that I can continue receiving updates to the SDK code.

If not, I guess I can write a program to do it myself. I would just rather have them maintain the code.

[JonathanHenson]

If you are static linking the executable wouldn’t this be done for you automatically by the linker?

…

Sent from my iPhone

On Nov 8, 2017, at 6:44 PM, Jon Smirl ***@***.***> wrote: I agree. I am hoping AWS will provide an automated way to strip out the unneeded definitions. I want it automated so that I can continue receiving updates to the SDK code. If not, I guess I can write a program to do it myself. I would just rather have them maintain the code. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

[jonsmirl]

I am statically linking and it does not happen with the linker. I believe that is due to a shared base class and virtual functions. It is not possible for the linker to be sure that the function is not used.

I also tried the scheme with '-fdata-sections -ffunction-sections -Wl,--gc-sections' and that did not eliminate them either.

If this was C code the linker would be able to eliminate them, with C++ it can't do it. Deleting them from the API definitions definitely works.

[marcomagdy]

There's definitely room for improvement here.
We need to conditionally mark the APIs as virtual via a compile-time definition. And we need to compile with -ffunction-sections -fdata-sections and link with -dead_strip.

In my local testing this brought down the size of the unit test executables by over 60%.

[crusader-mike]

(apologies if this isn't relevant):

C++ compilers can't eliminate unused virtual functions. I.e. if you reference a vtable -- you'll pull in all it's functions (and everything they reference). Old problem. GCC had this feature, but it wasn't working very well, so it was removed. Note that one class can have multiple vtables and even if function is overridden and never used -- it still gets pulled in (along with it's dependencies)

Edit: it goes worse if you use static variables -- once given translation unit is pulled in, all it's (global) static variables will be pulled in (followed by type's vtables and so forth)

[Skarlso]

This would be a great addition indeed. The linking around here is incredible even if only would like to use one service.

[u93]

Hi @jonsmirl , after reading thru this thread I realized that I'm having the same situation that you had a few years back... I'm working on an embedded system with a very small memory I'm trying to reduce the size of the AWS CPP SDK, which even after adding all the build options provided by AWS is still too big... Coincidently the set of features I'm looking for is similar to yours, I'm just using for now S3 PutObject and honestly do not need that huge amount of APIs built for S3...

I'm not an embedded developer, so this is proving to be quite the challenge for me... Do you still have samples around of what you did to reduce the SDK size?

Regards!

[jonsmirl]

I don't have access to the old project anymore, but the solution is to stop using the AWS SDK and make direct HTTPS calls using your own code. For S3 the simplest solution is to use pre-signed URLs.
https://docs.aws.amazon.com/AmazonS3/latest/userguide/ShareObjectPreSignedURL.html

[crusader-mike]

I suggest not using this SDK unless you need some specific non-trivial functionality it provides. We did a mistake of integrating it in our product couple of years ago and now are regretting it -- quite a few problems that we had to hack/workaround because fixes were talking too long. Some of those problems aren't fixed to this day. Now we have a Frankenstein in the codebase that is hard to upgrade because of these hacks -- it makes sense to simply invest a bit more and replace with a lightweight solution where we directly form HTTP requests ourselves and process responses.

…

On Mon, Jun 20, 2022, 3:52 PM Jon Smirl ***@***.***> wrote: I don't have access to the old project anymore, but the solution is to stop using the AWS and make direct HTTPS calls using your own code. For S3 the simplest solution is to use pre-signed URLs. https://docs.aws.amazon.com/AmazonS3/latest/userguide/ShareObjectPreSignedURL.html — Reply to this email directly, view it on GitHub <#724 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFKSIGFYNDIMJDUGLSEUAQLVQBZQ5ANCNFSM4EC5KXMQ> . You are receiving this because you commented.Message ID: ***@***.***>

[u93]

Thanks @jonsmirl and @crusader-mike ! I had the feeling that I was gonna have to do that... I do feel the same about the SDK, that is not worth it... Thanks for the great feedback, I really appreciate it!