KLayout DRC Performance - more than 10hrs for some user projects

[mkkassem]

Expected Behavior

user_project_wrapper: Runs under 1min :-)

Actual Behavior

user_project_wrapper: Takes 12mins to run the full DRC deck in flat mode

Steps to reproduce

Download the repo and use the run_drc.py command on the following GDS:
https://github.com/mabrains/globalfoundries-pdk-libs-gf180mcu_fd_pr/blob/main/rules/klayout/drc/run_drc.py
https://github.com/efabless/caravel_user_project/blob/gf180mcu/gds/user_project_wrapper.gds

Here is the run command:
python run_drc.py --gf180mcu=C --path=./user_project_wrapper.gds

Links to test cases

small: https://github.com/efabless/caravel_user_project/blob/gf180mcu/gds/user_project_wrapper.gds
medium: https://github.com/XDflight/ParaRAM/blob/c306db33edc38deed16b621a9c67fa48df8664f1/gds/user_project_wrapper.gds.gz
large: https://github.com/efabless/caravel-gf180mcu/blob/PnR/gds/caravel_core.gds.gz

Specifications

• Version: 0.27.11 or 0.27.12
• Platform: Ubuntu 22.04, Centos7 (Linux64)

[mkkassem]

@klayoutmatthias

[klayoutmatthias]

Thank you

[klayoutmatthias]

Intermediate data point:

With this PR #5 and the "wip" branch of KLayout (KLayout/klayout@1387498):

deep mode, no connectivity, one thread

CPU is i7-4800MQ, 2.7GHz

• small testcase: 96 s, memory 503 M
• medium testcase: 3502 s, memory 2234 M
• large testcase: 3768 s, memory 2761 M

[atorkmabrains]

@klayoutmatthias That's impressive.

[atorkmabrains]

cc @FaragElsayed2 @RehabSayed-G

[marwaneltoukhy]

Deep mode issues

Deep mode seems to break klayout, not sure if it's a python script issue or a klayout issue

command:

python3 run_drc.py --thr=150 --run_mode=deep --gf180mcu=C --path=/home/marwan/caravel-gf180mcu/gds/caravel_core.gds.gz

Error:

2022-11-30 11:37:58 -0800: Memory Usage (20832396K) : Executing rule DF.8_5V
addr2line: 'klayout': No such file
ERROR: Signal number: 11
Address: 0x198                                                                                                                                                                                                    
Program Version: KLayout 0.28 (2022-11-30 r5597ee896)                                                                                                                                                             
                                                                                                                                                                                                                  
Backtrace:                                                                                                                                                                                                        
/home/marwan/klayout/bin-release/libklayout_lay.so.0 +0x2bb93f lay::enable_signal_handler_gui(bool) [??:?]                                                                                                        
/lib/x86_64-linux-gnu/libpthread.so.0 +0x12730 __restore_rt [??:?]                                                                                                                                                
/home/marwan/klayout/bin-release/libklayout_db.so.0 +0x6e8a90 db::Layout::update() const [??:?]                                                                                                                   
/home/marwan/klayout/bin-release/libklayout_db.so.0 +0x62666d db::Cell::begin_parent_cells() const [??:?]                                                                                                         
/home/marwan/klayout/bin-release/libklayout_db.so.0 +0x114e266 db::local_processor<db::polygon_ref<db::polygon<int>, db::disp_trans<int> >, db::polygon_ref<db::polygon<int>, db::disp_trans<int> >, db::polygon_ref<db::polygon<int>, db::disp_trans<int> > >::compute_results(db::local_processor_contexts<db::polygon_ref<db::polygon<int>, db::disp_trans<int> >, db::polygon_ref<db::polygon<int>, db::disp_trans<int> >, db::polygon_ref<db::polygon<int>, db::disp_trans<int> > >&, db::local_operation<db::polygon_ref<db::polygon<int>, db::disp_trans<int> >, db::polygon_ref<db::polygon<int>, db::disp_trans<int> >, db::polygon_ref<db::polygon<int>, db::disp_trans<int> > > const*, std::vector<unsigned int, std::allocator<unsigned int> > const&) const [??:?]                                                                                        
/home/marwan/klayout/bin-release/libklayout_db.so.0 +0x114e890 db::local_processor<db::polygon_ref<db::polygon<int>, db::disp_trans<int> >, db::polygon_ref<db::polygon<int>, db::disp_trans<int> >, db::polygon_ref<db::polygon<int>, db::disp_trans<int> > >::run(db::local_operation<db::polygon_ref<db::polygon<int>, db::disp_trans<int> >, db::polygon_ref<db::polygon<int>, db::disp_trans<int> >, db::polygon_ref<db::polygon<int>, db::disp_trans<int> > >*, unsigned int, std::vector<unsigned int, std::allocator<unsigned int> > const&, std::vector<unsigned int, std::allocator<unsigned int> > const&) [??:?]                          
/home/marwan/klayout/bin-release/libklayout_db.so.0 +0x114eadf db::local_processor<db::polygon_ref<db::polygon<int>, db::disp_trans<int> >, db::polygon_ref<db::polygon<int>, db::disp_trans<int> >, db::polygon_ref<db::polygon<int>, db::disp_trans<int> > >::run(db::local_operation<db::polygon_ref<db::polygon<int>, db::disp_trans<int> >, db::polygon_ref<db::polygon<int>, db::disp_trans<int> >, db::polygon_ref<db::polygon<int>, db::disp_trans<int> > >*, unsigned int, unsigned int, std::vector<unsigned int, std::allocator<unsigned int> > const&) [??:?]                                                                             
/home/marwan/klayout/bin-release/libklayout_db.so.0 +0x1164e10 db::DeepRegion::selected_interacting_generic(db::Region const&, int, bool, db::InteractingOutputMode, unsigned long, unsigned long) const [??:?]   
/home/marwan/klayout/bin-release/libklayout_db.so.0 +0xdfc6c1 db::Region::selected_inside(db::Region const&) const [??:?]                                                                                         
/home/marwan/klayout/bin-release/libklayout_db.so.0 +0xe14d33 std::vector<db::Region, std::allocator<db::Region> >::~vector() [??:?]                                                                              
/home/marwan/klayout/bin-release/libklayout_rba.so.0 +0x26aea rba::RubyInterpreter::add_package_location(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) [??:?]           
/lib/x86_64-linux-gnu/libruby-2.5.so.2.5 +0x1b47a1 rb_iter_break_value [??:?]                                                                                                                                     
/lib/x86_64-linux-gnu/libruby-2.5.so.2.5 +0x1c1aeb rb_yield_block [??:?]                                                                                                                                          
/lib/x86_64-linux-gnu/libruby-2.5.so.2.5 +0x1c2084 rb_yield_block [??:?]                                                                                                                                          
/lib/x86_64-linux-gnu/libruby-2.5.so.2.5 +0x1ba306 rb_check_funcall [??:?]                                                                                                                                        
/lib/x86_64-linux-gnu/libruby-2.5.so.2.5 +0x1bdef4 rb_check_funcall [??:?]                                                                                                                                        
/lib/x86_64-linux-gnu/libruby-2.5.so.2.5 +0x1be9ef rb_check_funcall [??:?]                                                                                                                                        
/lib/x86_64-linux-gnu/libruby-2.5.so.2.5 +0x1c5c4b rb_mod_module_exec [??:?]                                                                                                                                      
/lib/x86_64-linux-gnu/libruby-2.5.so.2.5 +0x1b47a1 rb_iter_break_value [??:?]                                                                                                                                     
/lib/x86_64-linux-gnu/libruby-2.5.so.2.5 +0x1c238b rb_yield_block [??:?]                                                                                                                                          
/lib/x86_64-linux-gnu/libruby-2.5.so.2.5 +0x1ba306 rb_check_funcall [??:?]                                                                                                                                        
/lib/x86_64-linux-gnu/libruby-2.5.so.2.5 +0x1bdef4 rb_check_funcall [??:?]                                                                                                                                        
/lib/x86_64-linux-gnu/libruby-2.5.so.2.5 +0x1c256a rb_yield_block [??:?]                                                                                                                                          
/lib/x86_64-linux-gnu/libruby-2.5.so.2.5 +0x1c3687 rb_yield_block [??:?]                                                                                                                                          
/lib/x86_64-linux-gnu/libruby-2.5.so.2.5 +0x1c3e6e rb_funcallv [??:?]                                                                                                                                             
/lib/x86_64-linux-gnu/libruby-2.5.so.2.5 +0x97991 rb_protect [??:?]                                                                                                                                               
/home/marwan/klayout/bin-release/libklayout_rba.so.0 +0x45581 void std::vector<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, gsi::MethodBase const*>, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, gsi::MethodBase const*> > >::_M_realloc_insert<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, gsi::MethodBase const*> >(__gnu_cxx::__normal_iterator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, gsi::MethodBase const*>*, std::vector<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, gsi::MethodBase const*>, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, gsi::MethodBase const*> > > >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, gsi::MethodBase const*>&&) [??:?]                                                  
/home/marwan/klayout/bin-release/libklayout_rba.so.0 +0x48c44 tl::CancelException::CancelException() [??:?]                                                                                                       
/home/marwan/klayout/bin-release/libklayout_gsi.so.0 +0x9767f tl::event<gsi::ObjectBase::StatusEventType, void, void, void, void>::operator()(gsi::ObjectBase::StatusEventType) [??:?]                            
/home/marwan/klayout/bin-release/libklayout_gsi.so.0 +0x97943 tl::event<gsi::ObjectBase::StatusEventType, void, void, void, void>::operator()(gsi::ObjectBase::StatusEventType) [??:?]                            
/home/marwan/klayout/bin-release/libklayout_tl.so.0 +0xba938 tl::Executable::do_execute() [??:?]                                                                                                                  
/home/marwan/klayout/bin-release/libklayout_lym.so.0 +0x42d87 lym::MacroInterpreter::execute_macro(lym::Macro const*) [??:?]                                                                                      
/home/marwan/klayout/bin-release/libklayout_lym.so.0 +0x4cbc9 lym::Macro::run() const [??:?]                                                                                                                      
/home/marwan/klayout/bin-release/libklayout_lay.so.0 +0x1428b0 lay::ApplicationBase::run() [??:?]                                                                                                                 
klayout +0x4418                                                                                                                                                                                                   
/home/marwan/klayout/bin-release/libklayout_rba.so.0 +0x22760                                                                                                                                                     
/lib/x86_64-linux-gnu/libruby-2.5.so.2.5 +0x1b47a1                                                                                                                                                                
/lib/x86_64-linux-gnu/libruby-2.5.so.2.5 +0x1c238b                                                                                                                                                                
/lib/x86_64-linux-gnu/libruby-2.5.so.2.5 +0x1ba306                                                                                                                                                                
/lib/x86_64-linux-gnu/libruby-2.5.so.2.5 +0x1bdef4                                                                                                                                                                
/lib/x86_64-linux-gnu/libruby-2.5.so.2.5 +0x94f14                                                                                                                                                                 
/lib/x86_64-linux-gnu/libruby-2.5.so.2.5 +0x96dbd ruby_exec_node                                                                                                                                                  
/lib/x86_64-linux-gnu/libruby-2.5.so.2.5 +0x9926e ruby_run_node                                                                                                                                                   
/home/marwan/klayout/bin-release/libklayout_rba.so.0 +0x22c20 _ZN3rba15RubyInterpreter10initializeERiPPcPFiS1_S3_E                                                                                                
klayout +0x4097                                                                                                                                                                                                   
klayout +0x3a10                                                                                                                                                                                                   
/lib/x86_64-linux-gnu/libc.so.6 +0x2409b __libc_start_main                                                                                                                                                        
klayout +0x3a9a   

versions

klayout commit: 5597ee8960a8ad3f92fe9839cc822f483c1f5344
deck from Matthias fork

Improvements

Maybe when an error like this one occurs, klayout can report a more meaningful STDERR

[marwaneltoukhy]

Flat mode improvements

Running in flat mode on the large test took 6h48m, which is a huge improvement from the previous 16 to 18 hours.

command:

python3 run_drc.py --thr=150 --gf180mcu=C --path=/home/marwan/caravel-gf180mcu/gds/caravel_core.gds.gz

versions

klayout commit: 5597ee8960a8ad3f92fe9839cc822f483c1f5344
deck from Matthias fork

[klayoutmatthias]

@marwaneltoukhy I guess it is "thr=150" which breaks deep mode. Multiple cores do not really speed up in deep mode. For now, one thread is enough.

Please use commit KLayout/klayout@1387498 for wip branch, although I don't think that fixes the crash.

Matthias

[klayoutmatthias]

KLayout wip branch has been merged to master. You can use that one.

[atorkmabrains]

@klayoutmatthias when you are going to release that as an official release.

[klayoutmatthias]

My plan is end of next week. You can help by giving feedback. Like telling me if you encounter issues worth fixing before.

BTW: here are some datapoints for thread scaling (again deep mode, large test case, klayout master branch). I have only 4 cores, so I can't report performance scaling beyond that, but memory effect I can:

thr  Memory  Time
-------------------
1    2959M   3572s
2    3197M   3014s
4    3696M   2806s
8    4745M   ?
16   5859M   ?

So there is only a small effect and it is likely that this happens on checks that can be optimized anyway.

I'm using Python driver on Ubuntu 22.04:

  python3 run_drc.py --thr=$thr --run_mode=deep --gf180mcu=C --path=caravel_core.gds.gz

@marwaneltoukhy No crash observed. I don't have memory to run 150 threads in parallel, so my advise is to simply not use so many threads.

[atorkmabrains]

@klayoutmatthias will do some tests at my end. And provide feedback.

[klayoutmatthias]

@atorkmabrains Very good. Thanks. Not too many issues, please :)

[atorkmabrains]

@klayoutmatthias I have opened issues on the repo: https://github.com/mabrains/globalfoundries-pdk-libs-gf180mcu_fd_pr