New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

automatic recompilation of dependent functions #265

Closed
vtjnash opened this Issue Nov 26, 2011 · 55 comments

Comments

@vtjnash
Copy link
Member

vtjnash commented Nov 26, 2011

if I define a function d2 before a function d1 which calls d2 then change d2, d1 uses the old definition for d2.
I assume this is because it is all precompiled, but maybe there should be a note warning of this? Or would it be possible to replace the old definition with a longjmp to the new one?
(Mostly important for the REPL, since I don't always do a full load)

julia> function d2()
       a
       end

julia> function d()
         d2()
       end

julia> d()
in d: a not defined

julia> function d2()
       b=2
       end

julia> d()
in d: a not defined

julia> d2
Methods for generic function d2
d2() at prompt:2

julia> function d()
         d2()
       end

julia> d()
2
@StefanKarpinski

This comment has been minimized.

Copy link
Member

StefanKarpinski commented Nov 27, 2011

I believe this is because the code for d2() is generated when the function is defined so the d() method is resolved at that time. As you noted, you can redefine d2() as it was originally defined to make it work as expected. Ideally, we would automatically re-define anything that depends on d() when it is changed like this, but we don't have any of the metadata in place that would allow that. I guess we could put a warning about this behavior in the manual. Better still would be to fix it, but I think it's a bit tricky to do :-/

@ghost ghost assigned JeffBezanson Nov 28, 2011

@JeffBezanson

This comment has been minimized.

Copy link
Member

JeffBezanson commented Nov 28, 2011

I was hoping nobody would notice this, but I guess that wasn't realistic of me :)

@vtjnash

This comment has been minimized.

Copy link
Member

vtjnash commented Nov 28, 2011

It's more fun than that; it really lets you see the JIT at work, since it isn't resolved until execution:

julia> function f2()
       a
       end

julia> function f1()
       f2()
       end

julia> function f2()
       2
       end

julia> f1()
2

However, I fear this can lead to some unexpected results at the REPL, although I don't really know the best way to deal with it. Although I used an error to show the difference in dramatic fashion, it could also happens if I was trying to see the effect of changing an inner equation or constant!

@StefanKarpinski

This comment has been minimized.

Copy link
Member

StefanKarpinski commented Nov 28, 2011

There are two types of locations that may need to be updated when a method is altered:

  • places where the method is called explicitly as a function (call/ret)
  • places where the method has been inlined

The former could be updated just by changing the function body: it could be altered in place, or the calling sites could be actively updated to call a new location, or the old function body could be replaced with a stub that jumps to the new version, optionally patching the calling site. For inlining, we need to track the callers that have inlined the function and re-JIT them.

@JeffBezanson

This comment has been minimized.

Copy link
Member

JeffBezanson commented Nov 29, 2011

Really all we can do is discard code, because even if a method wasn't inlined any function that calls it might depend on type behavior that can change if the method changes. This is mostly for interactive cases so re-compiling code is not a huge deal.
Requires same work as issue #47.

@StefanKarpinski

This comment has been minimized.

Copy link
Member

StefanKarpinski commented Nov 29, 2011

Bummer. Well, since it's primarily the repl that's affected, doing the right thing slowly is fine. At run-time, this should almost never kick in — why would someone define something and then redefine it again except interactively? If the solution ends up inducing a lot of overhead maybe turning it one should be optional and automatically done in the repl?

@StefanKarpinski

This comment has been minimized.

Copy link
Member

StefanKarpinski commented Mar 15, 2012

We need to document that this is undefined currently.

@StefanKarpinski

This comment has been minimized.

Copy link
Member

StefanKarpinski commented Mar 25, 2012

Technically, this isn't bug, it's an undefined behavior. When you redefine a method, the resulting behavior is undefined. So Julia can do whatever it likes, including what it currently does. Providing a well-defined behavior upon method redefinition is a feature, not a bug fix. I'm also not convinced that this is a v1.0 issue since going from an undefined behavior to providing a well-defined behavior is not a breaking change. This could be implemented in v1.1 without breaking any valid v1.0 code.

@glycerine

This comment has been minimized.

Copy link

glycerine commented Apr 18, 2012

Greg Clayton from Apple's LLVM/LLDB staff was kind enough to document how to elicit (with lldb libraries, a subproject of llvm) the necessary information to determine a binary's dependencies from the embedded symbol info (symbol imports); as well as those symbols exported by a binary (necessary to construct the complete dependency graph).

  • Jason

    On Mar 31, 2012, at 11:02 PM, Jason E. Aten wrote:

    Dear LLDB enthusiasts,

    I'm wondering if I can use the lldb library/libraries to replace the certain code running on OSX that now returns two lists of symbols-- similar to the output of (dyldinfo -lazy_bind -exports ); i.e. I need to list the symbols imported and exported by a binary shared object or bundle.

    My hope was that by using an lldb library, I would be able to use the same client code on OSX as on linux. (The linux version of the code currently uses libbfd and libdld to do the same thing, but the later is getting little love/maintenance).

    I'm looking through include/lldb/, as it seems like lldb would need this same info (imported symbol list, and exported symbol list for a Mach-O file) to function, but it's not clear which API to use. All suggestions/pointers to example code in lldb would be welcome!

    Thank you.
    Jason

    In case it is unclear what dyldinfo does, here is an example: (but I only need the symbol names; not the addresses or segments or sections):

    $ file /tmp/sample_bundle
    /tmp/sample_bundle: Mach-O 64-bit bundle x86_64

    $dyldinfo -lazy_bind -export /tmp/sample_bundle

    lazy binding information (from lazy_bind part of dyld info):
    segment section address index dylib symbol
    __DATA __la_symbol_ptr 0x00001030 0x0000 flat-namespace __mm_pop_chunk
    __DATA __la_symbol_ptr 0x00001038 0x0015 flat-namespace _dh_define
    export information (from trie):
    0x000008A0 _C_ipair
    0x00000920 _init_ipair
    0x00000BC0 _C_iprot
    0x00000C40 _C_ipi2
    0x00000CC0 _C_ipi1
    0x00001040 _K_ipair_R43808f40
    0x00001160 _K_ipi1_R5cb4475d
    0x00001260 _K_ipi2_R5cb4475d
    0x00001360 _K_iprot_Rfc8fe739
    0x00001460 _majver_ipair
    0x00001464 _minver_ipair

On Mon, Apr 2, 2012 at 3:13 PM, Greg Clayton gclayton@apple.com wrote:

Yes you can do this with LLDB. If you load a binary and dump its symbol table, you will see the information you want. For symbols that are lazily bound, you can look for "Trampoline" symbols:

cd lldb/test/lang/objc/foundation
make
lldb a.out
(lldb) target modules dump symtab a.out
Symtab, file = .../lldb/test/lang/objc/foundation/a.out, num_symbols = 54:
              Debug symbol
              |Synthetic symbol
              ||Externally Visible
              |||
Index   UserID DSX Type         File Address/Value Load Address       Size               Flags      Name
------- ------ --- ------------ ------------------ ------------------ ------------------ ---------- ----------------------------------
[    0]      0 D   SourceFile   0x0000000000000000                    Sibling -> [   15] 0x00640000 /Volumes/work/gclayton/Documents/src/lldb/test/lang/objc/foundation/main.m
[    1]      2 D   ObjectFile   0x000000004f79f1ca                    0x0000000000000000 0x00660001 /Volumes/work/gclayton/Documents/src/lldb/test/lang/objc/foundation/main.o
[    2]      4 D   Code         0x00000001000010f0                    0x00000000000000c0 0x000e0000 -[MyString initWithNSString:]
[    3]      8 D   Code         0x00000001000011b0                    0x0000000000000090 0x000e0000 -[MyString dealloc]
[    4]     12 D   Code         0x0000000100001240                    0x00000000000000a0 0x000e0000 -[MyString description]
[    5]     16 D   Code         0x00000001000012e0                    0x0000000000000020 0x000e0000 -[MyString descriptionPauses]
[    6]     20 D   Code         0x0000000100001300                    0x0000000000000030 0x000e0000 -[MyString setDescriptionPauses:]
[    7]     24 D   Code         0x0000000100001330                    0x0000000000000030 0x000e0000 -[MyString str_property]
[    8]     28 D   Code         0x0000000100001360                    0x0000000000000050 0x000e0000 -[MyString setStr_property:]
[    9]     32 D   Code         0x00000001000013b0                    0x0000000000000040 0x000f0000 Test_Selector
[   10]     36 D   Code         0x00000001000013f0                    0x0000000000000130 0x000f0000 Test_NSString
[   11]     40 D   Code         0x0000000100001520                    0x0000000000000120 0x000f0000 Test_MyString
[   12]     44 D   Code         0x0000000100001640                    0x00000000000001b0 0x000f0000 Test_NSArray
[   13]     48 D   Code         0x00000001000017f0                    0x00000000000000e1 0x000f0000 main
[   14]     56 D X Data         0x0000000100002680                    0x0000000000000000 0x00200000 my_global_str
[   15]     58 D   SourceFile   0x0000000000000000                    Sibling -> [   19] 0x00640000 /Volumes/work/gclayton/Documents/src/lldb/test/lang/objc/foundation/my-base.m
[   16]     60 D   ObjectFile   0x000000004f79f1ca                    0x0000000000000000 0x00660001 /Volumes/work/gclayton/Documents/src/lldb/test/lang/objc/foundation/my-base.o
[   17]     62 D   Code         0x00000001000018e0                    0x0000000000000020 0x000e0000 -[MyBase propertyMovesThings]
[   18]     66 D   Code         0x0000000100001900                    0x000000000000001f 0x000e0000 -[MyBase setPropertyMovesThings:]
[   19]     82     Data         0x0000000100002000                    0x0000000000000460 0x000e0000 pvars
[   20]     83     ObjCIVar     0x0000000100002518                    0x0000000000000148 0x001e0000 MyBase.propertyMovesThings
[   21]     84   X Data         0x0000000100002660                    0x0000000000000008 0x000f0000 NXArgc
[   22]     85   X Data         0x0000000100002668                    0x0000000000000008 0x000f0000 NXArgv
[   23]     86   X ObjCClass    0x00000001000024d8                    0x0000000000000028 0x000f0000 MyBase
[   24]     87   X ObjCClass    0x0000000100002460                    0x0000000000000028 0x000f0000 MyString
[   25]     88   X ObjCIVar     0x0000000100002510                    0x0000000000000008 0x000f0000 MyString._desc_pauses
[   26]     89   X ObjCIVar     0x0000000100002508                    0x0000000000000008 0x000f0000 MyString.date
[   27]     90   X ObjCIVar     0x0000000100002500                    0x0000000000000008 0x000f0000 MyString.str
[   28]     91   X ObjCMetaClass 0x00000001000024b0                    0x0000000000000028 0x000f0000 MyBase
[   29]     92   X ObjCMetaClass 0x0000000100002488                    0x0000000000000028 0x000f0000 MyString
[   30]     97   X Data         0x0000000100002678                    0x0000000000000008 0x000f0000 __progname
[   31]     98   X Data         0x0000000100000000                    0x00000000000010b0 0x000f0010 _mh_execute_header
[   32]     99   X Data         0x0000000100002670                    0x0000000000000008 0x000f0000 environ
[   33]    101   X Data         0x0000000100002680                    0x0000000000000000 0x000f0000 my_global_str
[   34]    102   X Code         0x00000001000010b0                    0x0000000000000040 0x000f0000 start
[   35]    103     Trampoline   0x0000000100001938                    0x0000000000000006 0x00010200 NSLog
[   36]    104   X Undefined    0x0000000000000000                    0x0000000000000000 0x00010400 OBJC_CLASS_$_NSArray
[   37]    105   X Undefined    0x0000000000000000                    0x0000000000000000 0x00010200 OBJC_CLASS_$_NSAutoreleasePool
[   38]    106   X Undefined    0x0000000000000000                    0x0000000000000000 0x00010400 OBJC_CLASS_$_NSDate
[   39]    107   X Undefined    0x0000000000000000                    0x0000000000000000 0x00010400 OBJC_CLASS_$_NSObject
[   40]    108   X Undefined    0x0000000000000000                    0x0000000000000000 0x00010200 OBJC_CLASS_$_NSString
[   41]    109   X Undefined    0x0000000000000000                    0x0000000000000000 0x00010400 OBJC_METACLASS_$_NSObject
[   42]    110   X Undefined    0x0000000000000000                    0x0000000000000000 0x00010400 __CFConstantStringClassReference
[   43]    111   X Undefined    0x0000000000000000                    0x0000000000000000 0x00010100 _objc_empty_cache
[   44]    112   X Undefined    0x0000000000000000                    0x0000000000000000 0x00010100 _objc_empty_vtable
[   45]    113     Trampoline   0x000000010000193e                    0x0000000000000006 0x00010300 exit
[   46]    114     Trampoline   0x0000000100001920                    0x0000000000000006 0x00010100 objc_getProperty
[   47]    115     Trampoline   0x0000000100001926                    0x0000000000000006 0x00010100 objc_msgSend
[   48]    116     Trampoline   0x000000010000192c                    0x0000000000000006 0x00010100 objc_msgSendSuper2
[   49]    117   X Undefined    0x0000000000000000                    0x0000000000000000 0x00010100 objc_msgSend_fixup
[   50]    118     Trampoline   0x0000000100001932                    0x0000000000000006 0x00010100 objc_setProperty
[   51]    119     Trampoline   0x0000000100001944                    0x0000000000000006 0x00010300 printf
[   52]    120     Trampoline   0x000000010000194a                    0x0000000000000006 0x00010300 usleep
[   53]    121   X Undefined    0x0000000000000000                    0x0000000000000000 0x00010300 dyld_stub_binder
(lldb)


All lazily bound symbols will have type Trampoline:

[   45]    113     Trampoline   0x000000010000193e                    0x0000000000000006 0x00010300 exit
[   46]    114     Trampoline   0x0000000100001920                    0x0000000000000006 0x00010100 objc_getProperty
[   47]    115     Trampoline   0x0000000100001926                    0x0000000000000006 0x00010100 objc_msgSend
[   48]    116     Trampoline   0x000000010000192c                    0x0000000000000006 0x00010100 objc_msgSendSuper2
[   50]    118     Trampoline   0x0000000100001932                    0x0000000000000006 0x00010100 objc_setProperty
[   51]    119     Trampoline   0x0000000100001944                    0x0000000000000006 0x00010300 printf
[   52]    120     Trampoline   0x000000010000194a                    0x0000000000000006 0x00010300 usleep

The other symbols that are exernal are marked with an "X" (which is a boolean flag on each symbol).

The symbols can be accessed via the SBModule:

   size_t
   SBModule::GetNumSymbols ();

   lldb::SBSymbol
   SBModule::GetSymbolAtIndex (size_t idx);

And then you can get the symbol type from each SBSymbol:

   SymbolType
   SBSymbol::GetType ();


I just added the ability to see if a symbol is externally visible:
% svn commit
Sending        include/lldb/API/SBSymbol.h
Sending        scripts/Python/interface/SBSymbol.i
Sending        source/API/SBSymbol.cpp
Transmitting file data ...
Committed revision 153893.


   bool
   SBSymbol::IsExternal();


So your flow should be:

SBDebugger::Initialize();
SBDebugger debugger(SBDebugger::Create());
SBTarget target (debugger.CreateTarget (const char *filename,
                                       const char *target_triple,
                                       const char *platform_name,
                                       bool add_dependent_modules,
                                       lldb::SBError& error));

SBFileSpec exe_file_spec (filename);
SBModule exe_module (target.FindModule(exe_file_spec));
if (exe_module.IsValid()
{
   const size_t num_symbols = exe_module. GetNumSymbols();
   for (size_t i=0; i<num_symbols; ++i)
   {
       SBSymbol symbol (exe_module. GetSymbolAtIndex(i));
       if (symbol.IsExternal())
       {
       }

       if (symbol.GetType() == lldb::eSymbolTypeTrampoline)
       {
       }
   }
}




> _______________________________________________
> lldb-dev mailing list
> lldb-dev@cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/lldb-dev

On Mon, Apr 2, 2012 at 4:05 PM, Greg Clayton gclayton@apple.com wrote:

A quick clarification on the args to CreateTarget:

"filename" is just a full path to the local object file you want to observer. "target_triple" is your <arch>-<vendor>-<os>, or something like "x86_64-apple-darwin" or "i386-pc-linux" and can be NULL if the file is only a single architecture. "platform_name" can be NULL. "add_dependent_modules" should be false, since you are only interested in seeing the one object file itself, an SBError instance  should be created and passed in.
@StefanKarpinski

This comment has been minimized.

Copy link
Member

StefanKarpinski commented Mar 20, 2013

@staticfloat

This comment has been minimized.

Copy link
Member

staticfloat commented Jul 10, 2013

This is dredging up an amazingly old thread, but I realized I am possibly flirting around the edges of this problem with my codespeed benchmarking code. I repeatedly Core.include() files that contain two functions, listTests() and runTests(). I then simply call listTests() and runTests(), thereby redefining and then calling them every time I load a new benchmark file. Because I'm not redefining 2nd-tier functions I don't think I'll run into the problems listed here, but is redefining things like this a bad way to go about loading in multiple files with the same "API"?

@StefanKarpinski

This comment has been minimized.

Copy link
Member

StefanKarpinski commented Jul 10, 2013

Might be better to include them in a module repeatedly, but I realize that bumps up against the anonymous module problem that's also been raised on the dev list. In this case, it may be fine, however, since you can just define the same module multiple times and ignore the warning that emits.

@JeffBezanson

This comment has been minimized.

Copy link
Member

JeffBezanson commented Jul 10, 2013

Perhaps an old thread, but as relevant as ever.
Maybe another approach is to use evalfile and have the file "return" a tuple of functions. What's the anonymous module problem?

@StefanKarpinski

This comment has been minimized.

Copy link
Member

StefanKarpinski commented Jul 10, 2013

I was referring to this: #3661.

@jrevels jrevels referenced this issue May 23, 2017

Open

Add support for Julia v0.6 [Initial Build] #4

0 of 4 tasks complete

vtjnash added a commit that referenced this issue Jun 7, 2017

incremental serialize: restore recache order back to original
the commit fix for #265 reshuffled this order, but that was wrong
the correctness of jl_recache_other has since been fixed,
permitting this to go back to the correct original order

fix #22171

vtjnash added a commit that referenced this issue Jun 7, 2017

incremental serialize: restore recache order back to original
the commit fix for #265 reshuffled this order, but that was wrong
the correctness of jl_recache_other has since been fixed,
permitting this to go back to the correct original order

fix #22171

vtjnash added a commit that referenced this issue Jun 13, 2017

incremental serialize: restore recache order back to original
the commit fix for #265 reshuffled this order, but that was wrong
the correctness of jl_recache_other has since been fixed,
permitting this to go back to the correct original order

fix #22171

(cherry-picked from 8dab956, PR #22272)

vtjnash added a commit that referenced this issue Jun 15, 2017

incremental serialize: restore recache order back to original
the commit fix for #265 reshuffled this order, but that was wrong
the correctness of jl_recache_other has since been fixed,
permitting this to go back to the correct original order

fix #22171

(cherry-picked from 8dab956, PR #22272)

MikeInnes added a commit to JuliaLang/METADATA.jl that referenced this issue Sep 9, 2017

Upper bound StaticArrays versions (#9011)
* Upper bound StaticArrays versions

Notes:

* broadcast() changes broke some things early in 0.6 - I didn't track
  these down in detail, and have used 0.6- instead.

* Version 0.6.0-dev.1671 (44d7677) introduced the world changes to fix
  JuliaLang/julia#265, which broke most of the StaticArrays generated
  functions.

* StaticArrays-0.3.x is the maintainence branch for julia-0.5
  compatibility, and doesn't work with 0.6.0-dev.1671.  I didn't track
  down the upper bound here in detail.

* Fix julia upper bounds for Rotations 0.3.*

These are uninstallable on julia-0.6, due to the dependency on
StaticArrays and the new StaticArrays julia upper bounds.

@sglyon sglyon referenced this issue Sep 18, 2017

Closed

save to pdf #137

jrevels added a commit to jrevels/Cassette.jl that referenced this issue May 11, 2018

remove explicit world age propagation; turns out tests pass regardles…
…s as long as we reflect on world=typemax(UInt)

Note that this is not technically correct, since always treating world=typemax(UInt) is basically equivalent to reintroducing JuliaLang/julia#265. Thus, we still need to fix #6 eventually.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment