Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

coredump when set tf_enable_tao be true #1229

Open
houjincheng1992 opened this issue Jul 18, 2023 · 0 comments
Open

coredump when set tf_enable_tao be true #1229

houjincheng1992 opened this issue Jul 18, 2023 · 0 comments

Comments

@houjincheng1992
Copy link

tensorflow version is 1.15.5, cuda version cuda-10.1

when set TF_ENABLE_TAO true, coredump as following:

2023-07-17 21:34:47.554155: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:288] Restoring SavedModel bundle.
*** Aborted at 1689600887 (unix time) try "date -d @1689600887" if you are using GNU date ***
PC: @ 0x7f62463f39d4 tensorflow::tao::GetGlobalJitLevelForGraph()
*** SIGSEGV (@0x400000011) received by PID 288 (TID 0x7f6031358700) from PID 17; stack trace: ***
@ 0x7f629ffe0929 google::(anonymous namespace)::FailureSignalHandler()
@ 0x7f62a06a25f0 (unknown)
@ 0x7f62463f39d4 tensorflow::tao::GetGlobalJitLevelForGraph()
@ 0x7f624638fafa tensorflow::tao::(anonymous namespace)::MarkForCompilation()
@ 0x7f6246392fb0 tensorflow::tao::TaoMarkForCompilationPass::Run()
@ 0x7f624639403c tensorflow::tao::TaoOptimizationPass::Run()
@ 0x7f62b0ffb196 tensorflow::OptimizationPassRegistry::RunGrouping()
@ 0x7f62aeae25e0 tensorflow::GraphExecutionState::BuildGraph()
@ 0x7f62aea245da tensorflow::DirectSession::CreateGraphs()
@ 0x7f62aea25fc7 tensorflow::DirectSession::CreateExecutors()
@ 0x7f62aea29689 tensorflow::DirectSession::MakeCallable()
@ 0x7f62a7a3a95d tensorflow::(anonymous namespace)::RunOnce()
@ 0x7f62a7a3b56d tensorflow::(anonymous namespace)::RunRestore()
@ 0x7f62a7a3cdce tensorflow::(anonymous namespace)::LoadSavedModelInternal()
@ 0x7f62a7a3d8e4 tensorflow::LoadSavedModel()
@ 0x7f62a79eeffb tensorflow::serving::LoadSessionBundleOrSavedModelBundle()
@ 0x7f62a78d5bf5 tensorflow::serving::SavedModelBundleV2Factory::InternalCreateSavedModelBundleV2()
@ 0x7f62a78d6781 tensorflow::serving::SavedModelBundleV2Factory::CreateSavedModelBundleV2()
@ 0x7f62a78d1887 ZNSt17_Function_handlerIFN10tensorflow6StatusEPSt10unique_ptrINS0_18SavedModelBundleV2ESt14default_deleteIS3_EEEZNKS0_7serving31SavedModelBundleV2SourceAdapter18GetServableCreatorESt10shared_ptrINS9_25SavedModelBundleV2FactoryEERKSsEUlS7_E_E9_M_invokeERKSt9_Any_dataOS7
@ 0x7f62a78cf65e tensorflow::serving::SimpleLoader<>::Load()
@ 0x7f62a78b67b9 _ZNSt17_Function_handlerIFN10tensorflow6StatusEvEZNS0_7serving13LoaderHarness4LoadEvEUlvE_E9_M_invokeERKSt9_Any_data
@ 0x7f62a78b90ea tensorflow::serving::Retry()
@ 0x7f62a78b7ec6 tensorflow::serving::LoaderHarness::Load()
@ 0x7f62a78b3603 tensorflow::serving::BasicManager::ExecuteLoad()
@ 0x7f62a78b3b8c tensorflow::serving::BasicManager::ExecuteLoadOrUnload()
@ 0x7f62a78b56a2 tensorflow::serving::BasicManager::HandleLoadOrUnloadRequest()
@ 0x7f62a78b5892 _ZNSt17_Function_handlerIFvvEZN10tensorflow7serving12BasicManager20LoadOrUnloadServableERKNS3_19LoadOrUnloadRequestESt8functionIFvRKNS1_6StatusEEEEUlvE0_E9_M_invokeERKSt9_Any_data
@ 0x7f62b177be0c Eigen::ThreadPoolTempl<>::WorkerLoop()
@ 0x7f62b1778d98 _ZNSt17_Function_handlerIFvvEZN10tensorflow6thread16EigenEnvironment12CreateThreadESt8functionIS0_EEUlvE_E9_M_invokeERKSt9_Any_data
@ 0x7f629e1c7b9f execute_native_thread_routine
@ 0x7f62a069ae65 start_thread
@ 0x7f629dc2988d __clone

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant