Avoid Oracle OCCI APIs which work with std::string in C++11 ABI mode #17147
Conversation
We have enabled C++11 ABI in GCC 6.3.0 architecture. This means that things like std::string are not binary compatible between old "GCC 4" ABI and new one (C++11, default since GCC 5). The changes to GNU C++ Standard Library were needed to meet C++ Standard Library requirements for C++11. Note that GNU libstdc++ is dual-ABI. Oracle Instant Client is not compiled with latest GCC and C++11 ABI. One could avoid issues by using methods which do not use, e.g., std::string, or use C-level API. This lacks proper testings. All the tests within CMSSW are disabled for unknown reason. Also these are probably not used within IB RelVals. This does compile with GCC 6.3.0 + C++11 ABI. Signed-off-by: David Abdurachmanov <David.Abdurachmanov@cern.ch>
|
A new Pull Request was created by @davidlt for CMSSW_9_0_X. It involves the following packages: OnlineDB/CSCCondDB @ggovi, @cmsbuild, @davidlange6 can you please review it and eventually sign? Thanks. cms-bot commands are listed here #13028 |
|
please test |
|
The tests are being triggered in jenkins. |
|
Comparison job queued. |
|
ping^1 |
|
+1 |
cmsbuild
added
db-approved
fully-signed
and removed
db-pending
pending-signatures
labels
|
This pull request is fully signed and it will be integrated in one of the next CMSSW_9_0_X IBs (tests are also fine). This pull request requires discussion in the ORP meeting before it's merged. @davidlange6, @smuzaffar |
|
+1 |
|
This is a question to @davidlange6 . Also, could you provide full details on what kind of problem are you having? It's hard to understand what is happening from your original message. CC7 should not be a trigger as the same applies for SLC6. Thus there might be an extra element. Again, more details are needed to understand. |
|
here follows the error message we get: ----- Begin Fatal Exception 04-Sep-2017 11:24:38 CEST----------------------- The exception is thrown by the OCCI Environment::createConnection( … ) function. The trigger of the error is not the slc7, but the gcc630 compiler. |
|
Could you debug and confirm that ABI is causing this issue? |
|
Well with gcc530 it works... Thanks probably to |
|
any feedback on this? Can we restore the compilation option? |
|
Okay. So I had to investigate myself. I took random login credentials publicly floating on the internet (not gonna share here). Using CMSSW_9_2_0 + slc6_amd64_gcc630 + C++17 it failed. Using As mentioned in original comment I would highly suggest to use C-level API as that is safe. You can also use |
|
The OCCI choice is non-standard in cms, the recommended interface is coral, which is based on OCI. So, for whoever is currently using OCCI in CMSSW, the possible fixes are obviously the two you are mentioning:
Option 2. looks cheaper, I would be in favour of it. If there are reasons not to go for it, please let us know. |
|
I guess one relevant point could be the medium term availability ( for Run2 ) of this option in the compilers we use. Is it available in gcc700? Can you please check if it works? |
|
You mean Note that this is not magic bullet and you need to use it accordingly, refactoring of some code might be needed. This flag control ABI within GNU Standard C++ Library and is used mainly to build libraries with dual-ABI. |
|
Let me make a step back on this. We have some CMSSW code that does not work since we moved to gcc630. We know the reason, we have an optimal solution ( get rid of OCCI ), but it requires some work. Is the workaround used in gcc530 still a solution until the end of run2? If for any reason related to the planning (in terms of compilers ) this is not the case, we need to tell to the owners of the affected packages that they have to migrate to OCI or coral. |
|
It would be great if "no". What work is involved to make this happen for 2018 data taking release
Cheers,
David
On 30 Oct 2017, at 20:24, ggovi <notifications@github.com<mailto:notifications@github.com>> wrote:
Let me make a step back on this. We have some CMSSW code that does not work since we moved to gcc630. We know the reason, we have an optimal solution ( get rid of OCCI ), but it requires some work. Is the workaround used in gcc530 still a solution until the end of run2? If for any reason related to the planning (in terms of compilers ) this is not the case, we need to tell to the owners of the affected packages that they have to migrate to OCI or coral.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub<#17147 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AEzywwrb00tDiVbvlQqe1ByOzMddDogVks5sxiJMgaJpZM4Lgo2Q>.
|
|
I've asked the people to provide an estimation... |
|
Hi, |
|
Hi @ggovi
Probably a good topic for a future alca meeting. what is the relationship/dependency of CMSSW and this other large code base? it depends on CMSSW?
… On Nov 8, 2017, at 12:09 PM, ggovi ***@***.***> wrote:
Hi,
the Ecal team already started to work on the migration. The SiStrip team looks in a more complex situation:
OCCI is used in 45 classes and ~30k lines of code are involved in a package external to CMSSW.
Should we invite them to some discussion ( ORP or a dedicate meeting )?
Please let me know...
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.
|
|
Hi @davidlange6 The external package for SiStrip that uses OCCI is tkonline. It is part of the trackerDAQ software and is compiled as a standalone package online (without depending on CMSSW). Inside CMSSW, it is mainly used to access the online SiStrip configuration database for e.g., the Strips O2O, etc. |
|
ok, so the issue is not the entire code base of tkonlinesw, but the piece of it needed as a dependency of cmssw for the strips O2O (which is apparently currently silenctly broken..) Thanks.
… On Nov 8, 2017, at 3:34 PM, hqucms ***@***.***> wrote:
Hi @davidlange6
The external package for SiStrip that uses OCCI is tkonline. It is part of the trackerDAQ software and is compiled as a standalone package online (without depending on CMSSW). Inside CMSSW, it is mainly used to access the online SiStrip configuration database for e.g., the Strips O2O, etc.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.
|
|
None of RelVals use that functionality thus not so visible. In general it's only use in ONLINE. E.g, AArch64 is using a fake tkonline (https://github.com/cms-sw/cmsdist/blob/IB/CMSSW_10_0_X/gcc630/tkonlinesw-fake.spec) which just provides the interfaces needed by CMSSW to compile. If by accident is something called you would received exception saying that implementation is missing. There seems to be a number of functions (not sure if public) which use thing like |
|
that is: relvals, add-on tests, unit tests, etc...which is maybe a bad thing.. such brokenness should be harder to ignore..
… On Nov 8, 2017, at 3:55 PM, davidlt ***@***.***> wrote:
None of RelVals use that functionality thus not so visible. In general it's only use in ONLINE. E.g, AArch64 is using a fake tkonline (https://github.com/cms-sw/cmsdist/blob/IB/CMSSW_10_0_X/gcc630/tkonlinesw-fake.spec) which just provides the interfaces needed by CMSSW to compile. If by accident is something called you would received exception saying that implementation is missing.
There seems to be a number of functions (not sure if public) which use thing like oracle::occi::Clob, std::string, etc.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.
|
|
Hi @davidlt A naive question: I was wondering if it would be possible to patch tkonline somehow so that it does not communicate with the outside using |
|
It's enough to use C primitives instead of C++ standard classes to avoid C++ standard library ABI changes. Thus E.g., I am not sure if |
|
Thank you @davidlt ! I will take a look! |
|
The tracker online team found a way to work around the failure with the following piece of code: extern "C" {
#include <string.h>
int _ZNKSs6lengthEv (char **a)
{
char *s = *a;
return strlen (s);
}
}One just needs to compile it into a shared library and preload it with Basically it is replacing the code for Any comments on this, @davidlt @davidlange6 ? |
|
I see how this could work, but I am not (yet) fully sure if it's safe in all cases. I would probably prefer not to have it within CMSSW repository. |
|
@hqucms |
|
@davidlt @davidlange6 @hqucms |
|
@ggovi Sure, I will ask them now. |
|
@davidlt @davidlange6 @ggovi @boudoul I just had a quick look at the new version of OCCI 12.2.0.1.0. Indeed the symbol ------ More details below ------ A simple example for reproducing it: #include <iostream>
#include <string>
#include <sstream>
#include "occi.h"
using namespace oracle::occi;
extern "C" {
#include <string.h>
int _ZNKSs6lengthEv (char **a)
{
char *s = *a;
return strlen (s);
}
}
int main(){
std::string user = "a";
std::string pass = "b";
std::string sid = "c";
auto env = Environment::createEnvironment(Environment::OBJECT);
auto conn = env->createConnection(user, pass, sid);
auto stmt = conn->createStatement();
return 0;
}Compiling e.g., g++ occi.cc -I/cvmfs/cms.cern.ch/slc6_amd64_gcc630/external/oracle/12.1.0.2.0/include -L/cvmfs/cms.cern.ch/slc6_amd64_gcc630/external/oracle/12.1.0.2.0/lib -locci -lclntshWith OCCI 12.1.0.2.0, it gives while in the new OCCI 12.2.0.1.0, we are back to the original error related to the string length: |
We have enabled C++11 ABI in GCC 6.3.0 architecture. This means that
things like std::string are not binary compatible between old "GCC 4"
ABI and new one (C++11, default since GCC 5). The changes to GNU C++
Standard Library were needed to meet C++ Standard Library requirements
for C++11. Note that GNU libstdc++ is dual-ABI.
Oracle Instant Client is not compiled with latest GCC and C++11 ABI. One
could avoid issues by using methods which do not use, e.g., std::string,
or use C-level API.
This lacks proper testings. All the tests within CMSSW are disabled for
unknown reason. Also these are probably not used within IB RelVals.
This does compile with GCC 6.3.0 + C++11 ABI.
Signed-off-by: David Abdurachmanov David.Abdurachmanov@cern.ch