support python `with` keyword #8

chanshunli · 2019-11-14T05:17:26Z

PyCall.jl pywith

@pywith pybuiltin("open")("file.txt","w") as f begin
    f.write("hello")
end

cnuernber · 2019-11-14T22:11:31Z

here is 'with' documentation.

Something like:

user>  (defmacro with 
        [bind-vec & body]
         (let [varnames (map first (partition 2 bind-vec))]
           `(let [~@bind-vec]
              (try 
                ~@(->> varnames
                       (map (fn [var-sym]
                              `(py/call-attr ~var-sym "__enter__"))))
              ~@body
              (finally 
                 ~@(->> varnames
                        (map (fn [var-sym]
                               `(py/call-attr ~var-sym "__exit__")))))))))
           
#'user/with
user> (with [f (py/call-attr io "open" "project.clj")]
            (py/call-attr f "readlines"))
            
['(defproject cnuernber/libpython-clj "1.9-SNAPSHOT"\n', '  :description "libpython bindings to the techascent ecosystem"\n', '  :url "http://github.com/cnuernber/libpython-clj"\n', '  :license {:name "EPL-2.0"\n', '            :url "https://www.eclipse.org/legal/epl-2.0/"}\n', '  :dependencies [[org.clojure/clojure "1.10.1"]\n', '                 [techascent/tech.datatype "4.46"]\n', '                 [camel-snake-kebab "0.4.0"]]\n', '  :repl-options {:init-ns user}\n', '  :java-source-paths ["java"])\n']
user> (macroexpand '(with [f (py/call-attr io "open" "project.clj")]
            (py/call-attr f "readlines")))
(let*
 [f (py/call-attr io "open" "project.clj")]
 (try
  (libpython-clj.python/call-attr f "__enter__")
  (py/call-attr f "readlines")
  (finally (libpython-clj.python/call-attr f "__exit__"))))

cnuernber · 2019-11-14T22:27:34Z

Thumbs up on the Julia interest, btw :-).

chanshunli · 2019-11-15T01:45:14Z

Wow，Very Cool ! ! ! Thanks Chris

chanshunli · 2019-11-15T02:08:04Z

(defonce autograd (py-import "mxnet.autograd"))
(defonce nd (-a> mxnet nd))

(defmacro $ [py-import-lib attr-name & args]
  `(if (= (get (System/getenv) "LEIN_TYPE") "FRONTEND")
     nil
     (call-attr
      ~py-import-lib  ~(str attr-name) ~@args)))

;; (-a> ($ np ones [2 3]) shape) ;; => (2, 3)
(defmacro -a> [py-import-lib attr-name]
  `(if (= (get (System/getenv) "LEIN_TYPE") "FRONTEND")
     nil
     (get-attr
      ~py-import-lib  ~(str attr-name))))

  ;; x = nd.arange(4).reshape((4, 1))
  (def x (-> nd
             ($ arange 4)
             ($ reshape [4 1])))

  ($ x attach_grad)

  ;; 2 * nd.dot(x.T, x)
  (with [record ($ autograd record)]
        (def y ($ nd multiply 2
                  ($ nd dot (-a> x T) x)))
        ($ y backward))

cnuernber · 2019-11-15T02:15:01Z

Ahhh, the joys of native library development! Takes me back to my days as a much younger man...

Could you attach hs_err_pid12832.log? Seems odd that strlen would crash anything, maybe an error happened and the error reporting system is then causing a crash.

chanshunli · 2019-11-15T02:19:44Z

the python is here:


In [1]: from mxnet import autograd, nd                                                       

In [2]: x = nd.arange(4).reshape((4, 1))                                                     

In [3]: x.attach_grad()                                                                      

In [4]: with autograd.record(): 
   ...:     y = 2 * nd.dot(x.T, x) 
   ...:     y.backward() 
   ...:                                                                                      

In [6]: x.grad                                                                               
Out[6]: 

[[ 0.]
 [ 4.]
 [ 8.]
 [12.]]
<NDArray 4x1 @cpu(0)>

In [7]:

chanshunli · 2019-11-15T02:24:37Z

@cnuernber I have sent the log file to your mailbox chris@techascent.com.

Environment Variables:
CLASSPATH=/Users/clojure/.lein/self-installs/leiningen-2.9.1-standalone.jar
SHELL=/bin/zsh

Signal Handlers:
SIGSEGV: [libjvm.dylib+0x5b1141], sa_mask[0]=11111111011111110111111111111111, sa_flags=SA_ONSTACK|SA_RESTART|SA_SIGINFO
SIGBUS: [libjvm.dylib+0x5b1141], sa_mask[0]=11111111011111110111111111111111, sa_flags=SA_RESTART|SA_SIGINFO
SIGFPE: [libjvm.dylib+0x487bc4], sa_mask[0]=11111111011111110111111111111111, sa_flags=SA_RESTART|SA_SIGINFO
SIGPIPE: SIG_IGN, sa_mask[0]=00000000000000000000000000000000, sa_flags=SA_RESTART
SIGXFSZ: SIG_IGN, sa_mask[0]=00000000000000000000000000000000, sa_flags=none
SIGILL: [libjvm.dylib+0x487bc4], sa_mask[0]=11111111011111110111111111111111, sa_flags=SA_RESTART|SA_SIGINFO
SIGUSR1: SIG_DFL, sa_mask[0]=11011111011111100000000111000110, sa_flags=none
SIGUSR2: [libjvm.dylib+0x4876e2], sa_mask[0]=00100000000000000000000000000000, sa_flags=SA_RESTART|SA_SIGINFO
SIGHUP: [libjvm.dylib+0x485cb9], sa_mask[0]=11111111011111110111111111111111, sa_flags=SA_RESTART|SA_SIGINFO
SIGINT: [libjvm.dylib+0x485cb9], sa_mask[0]=11111111011111110111111111111111, sa_flags=SA_RESTART|SA_SIGINFO
SIGTERM: [libjvm.dylib+0x485cb9], sa_mask[0]=11111111011111110111111111111111, sa_flags=SA_RESTART|SA_SIGINFO
SIGQUIT: [libjvm.dylib+0x485cb9], sa_mask[0]=11111111011111110111111111111111, sa_flags=SA_RESTART|SA_SIGINFO


---------------  S Y S T E M  ---------------

OS:Bsduname:Darwin 19.0.0 Darwin Kernel Version 19.0.0: Thu Oct 17 16:17:15 PDT 2019; root:xnu-6153.41.3~29/RELEASE_X86_64 x86_64
rlimit: STACK 9788k, CORE 0k, NPROC 1392, NOFILE 10240, AS infinity
load average:1.88 1.95 2.25

CPU:total 4 (2 cores per cpu, 2 threads per core) family 6 model 61 stepping 4, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3, sse4.1, sse4.2, popcnt, avx, avx2, aes, clmul, erms, 3dnowpref, lzcnt, ht, tsc, tscinvbit, bmi1, bmi2, adx

Memory: 4k page, physical 8388608k(218156k free)

/proc/meminfo:


vm_info: Java HotSpot(TM) 64-Bit Server VM (25.121-b13) for bsd-amd64 JRE (1.8.0_121-b13), built on Dec 12 2016 20:39:32 by "java_re" with gcc 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2336.11.00)

time: Fri Nov 15 10:04:40 2019
elapsed time: 348 seconds (0d 0h 5m 48s)

chanshunli · 2019-11-15T16:29:56Z

@cnuernber Have you received my mail yet? Thanks Chris :-D

cnuernber · 2019-11-15T16:40:54Z

Yep, got it. There error happens when libpython-clj detects an error in the underlying code in a call-attr call. it tries to print it out and then bang. My guess is there is some context for mxnet that needs to be setup that isn't getting done.

Stack: [0x00007000066f7000,0x00007000067f7000],  sp=0x00007000067f3e40,  free space=1011k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C  [libsystem_platform.dylib+0x1732]  _platform_strlen+0x12
C  [Python+0x790ac]  unicode_fromformat_write_cstr+0x1f
C  [Python+0x61010]  PyUnicode_FromFormatV+0x69d
C  [Python+0xbde11]  PyErr_FormatV+0x29
C  [Python+0xbd293]  PyErr_Format+0x84
C  [Python+0x46461]  PyObject_GetAttr+0x99
C  [Python+0x27430]  PyFile_WriteObject+0x27
C  [Python+0x27526]  PyFile_WriteString+0x45
C  [Python+0xd5d76]  print_exception_recursive+0x53c
C  [Python+0xd5797]  PyErr_Display+0x6e
C  [Python+0xdda51]  sys_excepthook+0x4e
C  [Python+0x19d4d]  _PyMethodDef_RawFastCallDict+0xbf
C  [Python+0x194aa]  _PyCFunction_FastCallDict+0x2c
C  [Python+0xd5462]  PyErr_PrintEx+0x11b
C  [jna5219763700594524346.tmp+0xde74]  ffi_call_unix64+0x4c
C  0x00007000067f4ca8

Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
J 6781  com.sun.jna.Native.invokeVoid(Lcom/sun/jna/Function;JI[Ljava/lang/Object;)V (0 bytes) @ 0x0000000112b02729 [0x0000000112b026c0+0x69]
J 6690 C1 com.sun.jna.Function.invoke([Ljava/lang/Object;Ljava/lang/Class;ZI)Ljava/lang/Object; (769 bytes) @ 0x000000011418039c [0x000000011417b860+0x4b3c]
J 6694 C1 com.sun.jna.Function.invoke(Ljava/lang/reflect/Method;[Ljava/lang/Class;Ljava/lang/Class;[Ljava/lang/Object;Ljava/util/Map;)Ljava/lang/Object; (548 bytes) @ 0x00000001135ffc0c [0x00000001135fe2c0+0x194c]
J 6689 C1 com.sun.jna.Function.invoke(Ljava/lang/Class;[Ljava/lang/Object;Ljava/util/Map;)Ljava/lang/Object; (41 bytes) @ 0x000000011336adfc [0x000000011336a900+0x4fc]
J 6780 C1 com.sun.jna.Function.invoke([Ljava/lang/Object;)V (9 bytes) @ 0x0000000112a418b4 [0x0000000112a417c0+0xf4]
j  libpython_clj.jna.concrete.err$PyErr_Print.invokeStatic()Ljava/lang/Object;+56
j  libpython_clj.jna.concrete.err$PyErr_Print.invoke()Ljava/lang/Object;+0
J 6735 C1 clojure.lang.Var.invoke()Ljava/lang/Object; (10 bytes) @ 0x00000001133d9154 [0x00000001133d8ce0+0x474]
j  libpython_clj.python.interpreter$check_error_str$fn__56102$fn__56103.invoke()Ljava/lang/Object;+11
J 4306 C2 clojure.lang.AFn.applyToHelper(Lclojure/lang/IFn;Lclojure/lang/ISeq;)Ljava/lang/Object; (3238 bytes) @ 0x00000001135c4f3c [0x00000001135c4380+0xbbc]
J 3884 C2 clojure.lang.AFn.applyTo(Lclojure/lang/ISeq;)Ljava/lang/Object; (12 bytes) @ 0x000000011349f014 [0x000000011349efe0+0x34]
J 6832 C1 clojure.core$with_bindings_STAR_.doInvoke(Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object; (16 bytes) @ 0x0000000113332abc [0x0000000113332540+0x57c]
J 4882 C2 clojure.lang.RestFn.invoke(Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object; (119 bytes) @ 0x0000000113bd1ab0 [0x0000000113bd18e0+0x1d0]
j  libpython_clj.python.interpreter$check_error_str$fn__56102.invoke()Ljava/lang/Object;+67
j  libpython_clj.python.interpreter$with_gil_fn.invokeStatic(Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object;+465
J 6661 C1 libpython_clj.python.interpreter$with_gil_fn.invoke(Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object; (10 bytes) @ 0x000000011326f7bc [0x000000011326f740+0x7c]
j  libpython_clj.python.interpreter$check_error_str.invokeStatic()Ljava/lang/Object;+17
j  libpython_clj.python.interpreter$check_error_str.invoke()Ljava/lang/Object;+0
j  libpython_clj.python.interpreter$check_error_throw.invokeStatic()Ljava/lang/Object;+9
j  libpython_clj.python.interpreter$check_error_throw.invoke()Ljava/lang/Object;+0
j  libpython_clj.python.object$wrap_pyobject.invokeStatic(Ljava/lang/Object;)Ljava/lang/Object;+9
j  libpython_clj.python.object$wrap_pyobject.invoke(Ljava/lang/Object;)Ljava/lang/Object;+3
j  libpython_clj.python.object$eval58298$fn__58299$fn__58300.invoke()Ljava/lang/Object;+204
j  libpython_clj.python.interpreter$with_gil_fn$fn__56071.invoke()Ljava/lang/Object;+28
J 4306 C2 clojure.lang.AFn.applyToHelper(Lclojure/lang/IFn;Lclojure/lang/ISeq;)Ljava/lang/Object; (3238 bytes) @ 0x00000001135c4f3c [0x00000001135c4380+0xbbc]
J 3884 C2 clojure.lang.AFn.applyTo(Lclojure/lang/ISeq;)Ljava/lang/Object; (12 bytes) @ 0x000000011349f014 [0x000000011349efe0+0x34]
J 6832 C1 clojure.core$with_bindings_STAR_.doInvoke(Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object; (16 bytes) @ 0x0000000113332abc [0x0000000113332540+0x57c]
J 4882 C2 clojure.lang.RestFn.invoke(Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object; (119 bytes) @ 0x0000000113bd1ab0 [0x0000000113bd18e0+0x1d0]
j  libpython_clj.python.interpreter$with_gil_fn.invokeStatic(Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object;+110
J 6661 C1 libpython_clj.python.interpreter$with_gil_fn.invoke(Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object; (10 bytes) @ 0x000000011326f7bc [0x000000011326f740+0x7c]
j  libpython_clj.python.object$eval58298$fn__58299.invoke(Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object;+28
j  libpython_clj.python.protocols$eval56462$fn__56463$G__56453__56472.invoke(Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object;+60
j  libpython_clj.python.bridge$generic_python_as_jvm$fn$reify__58812.do_call_fn(Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object;+50
j  libpython_clj.python.protocols$call_attr.invokeStatic(Ljava/lang/Object;Ljava/lang/Object;Lclojure/lang/ISeq;)Ljava/lang/Object;+106
j  libpython_clj.python.protocols$call_attr.doInvoke(Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object;+12
J 4882 C2 clojure.lang.RestFn.invoke(Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object; (119 bytes) @ 0x0000000113bd1ab0 [0x0000000113bd18e0+0x1d0]
J 6955 C1 clojure.lang.Var.invoke(Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object; (24 bytes) @ 0x0000000112b7c6d4 [0x0000000112b7c1c0+0x514]
j  luminus_shadow_txt2code.libpython.mxnet$eval65229.invokeStatic()Ljava/lang/Object;+256
j  luminus_shadow_txt2code.libpython.mxnet$eval65229.invoke()Ljava/lang/Object;+0

chanshunli · 2019-11-15T17:15:26Z

I don't know what to do after executing with autograd.record(): {code} , what did it do after the end.
Remove the with method, I don't know how to write in the mxnet.

cnuernber · 2019-11-15T17:48:15Z

One thing that could be going on is the with statement returns the last thing. The exit happens and then the repl attempts to print the last thing that was done within the context of the with. But the record has been exited by that time and this can cause a crash. In those cases I would return :ok as the last statement in the with to avoid that chance. I noticed in the python mxnet example they don't ever access variables defined within the record outside of the record.

chanshunli · 2019-11-15T17:53:39Z

How to use the gdb debug it, What happened inside with autograd.record(): {code} ?

like this:

 /usr/local/bin/gdb -args /usr/local/Cellar/python/3.7.1/Frameworks/Python.framework/Versions/3.7/bin/python  /Users/clojure/Desktop/hack_mxnet.py --core ./core

from mxnet import autograd, nd

x = nd.arange(4).reshape((4, 1))

x.attach_grad()

print('before with autograd.record')

with autograd.record():
    print('autograd.record ing ...')
    y = 2 * nd.dot(x.T, x)
    y.backward()

print('after with autograd.record')


print(x.grad)

chanshunli · 2019-11-15T18:17:35Z

@cnuernber Do you mean this? => I got this Error.

  ;; x = nd.arange(4).reshape((4, 1))
  (def x (-> nd
             ($ arange 4)
             ($ reshape [4 1])))

  ($ x attach_grad)

  ;; 2 * nd.dot(x.T, x)
  (with [record ($ autograd record)]
        (do (let [y ($ nd multiply 2
                       ($ nd dot (-a> x T) x))]
              ($ y backward))
            "ok"))

cnuernber · 2019-11-15T18:44:27Z

Yes, that is what I meant.

The documentation link I posted above hints at what is going on. exit takes 3 additional positional arguments that I believe relate to the current in-flight exception. This will take more work to make solid, but you could modify the macro to call (py/call-attr ~argval "exit" nil nil nil) for now to see how that might work.

chanshunli · 2019-11-15T18:50:58Z

Although with {...} went wrong, x.grad is right. When I repl the with {...} , the program always crashes and exits.

chanshunli · 2019-11-15T19:24:29Z

Thanks @cnuernber Chris, It's work good 👍 👍 👍

(defmacro with
  [bind-vec & body]
  (let [varnames (map first (partition 2 bind-vec))]
    `(let [~@bind-vec]
       (try
         ~@(->> varnames
                (map (fn [var-sym]
                       `(py/call-attr ~var-sym "__enter__"))))
         ~@body
         (finally
           ~@(->> varnames
                  (map (fn [var-sym]
                         `(py/call-attr ~var-sym "__exit__" nil nil nil)))))))))

(comment

  ;; x = nd.arange(4).reshape((4, 1))
  (def x (-> nd
             ($ arange 4)
             ($ reshape [4 1])))

  ($ x attach_grad)

  ;; 2 * nd.dot(x.T, x)
  (with [record ($ autograd record)]
        (let [y ($ nd multiply 2
                   ($ nd dot (-a> x T) x))]
          ($ y backward))
        )

  (def dx (-a> x grad))
  ;; =>
;; [[ 0.]
;;  [ 4.]
;;  [ 8.]
;;  [12.]]
;; <NDArray 4x1 @cpu(0)>

  ;;x = mx.nd.random.uniform(shape=(10,))
  (def x1 (-> mxnet
              (-a> nd)
              (-a> random)
              (call-attr-kw "uniform" [] {"shape" 10})))

  ($ x1 attach_grad)

  (with [record ($ autograd record)]
        (let [m ($ nd sigmoid x1)]
          ($ m backward)))

  (def dx1 (-a> x1 grad)) ;; [0. 0. 0. 0. 0. 0. 0. 0. 0. 0.] =>  [0.23208144 0.22925895 0.2205717  0.21025445 0.22859976 0.20910253 0.23232493 0.2100039  0.23910952 0.22719035]

  )

cnuernber · 2019-11-15T21:21:33Z

goooaaaallll!!!!!

That is awesome :-)!!

👍 💯

chanshunli · 2019-11-16T01:23:52Z

@cnuernber 🥇 💯 👍 Can you add with macro to the cnuernber/libpython-clj library?

cnuernber · 2019-11-16T15:11:35Z

I will eventually. It interacts with error handling so the 'correct' fix will involve some thinking.

cnuernber · 2019-11-17T19:54:19Z

https://github.com/cnuernber/libpython-clj/blob/master/example/src/mxnet_autograd.clj

with is now included in python.clj. And tested in the unit tests :-). Thank you for this issue, it rocked :-)!!

chanshunli · 2019-11-18T04:05:38Z

@cnuernber Good Job 👍 👍 👍 💯 💯 💯 https://clojars.org/cnuernber/libpython-clj , I am updating to the 1.9 version right now.

cnuernber · 2019-11-18T17:33:18Z

👍 - best issue ever :-).

cnuernber added enhancement New feature or request good first issue Good for newcomers labels Nov 14, 2019

cnuernber added harder Harder issue and removed good first issue Good for newcomers labels Nov 15, 2019

cnuernber closed this as completed Nov 17, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

support python `with` keyword #8

support python `with` keyword #8

chanshunli commented Nov 14, 2019

cnuernber commented Nov 14, 2019

cnuernber commented Nov 14, 2019

chanshunli commented Nov 15, 2019

chanshunli commented Nov 15, 2019

cnuernber commented Nov 15, 2019

chanshunli commented Nov 15, 2019

chanshunli commented Nov 15, 2019

chanshunli commented Nov 15, 2019

cnuernber commented Nov 15, 2019

chanshunli commented Nov 15, 2019

cnuernber commented Nov 15, 2019

chanshunli commented Nov 15, 2019

chanshunli commented Nov 15, 2019

cnuernber commented Nov 15, 2019

chanshunli commented Nov 15, 2019

chanshunli commented Nov 15, 2019

cnuernber commented Nov 15, 2019

chanshunli commented Nov 16, 2019

cnuernber commented Nov 16, 2019

cnuernber commented Nov 17, 2019

chanshunli commented Nov 18, 2019

cnuernber commented Nov 18, 2019

support python with keyword #8

support python with keyword #8

Comments

chanshunli commented Nov 14, 2019

cnuernber commented Nov 14, 2019

cnuernber commented Nov 14, 2019

chanshunli commented Nov 15, 2019

chanshunli commented Nov 15, 2019

cnuernber commented Nov 15, 2019

chanshunli commented Nov 15, 2019

the python is here:

chanshunli commented Nov 15, 2019

chanshunli commented Nov 15, 2019

cnuernber commented Nov 15, 2019

chanshunli commented Nov 15, 2019

cnuernber commented Nov 15, 2019

chanshunli commented Nov 15, 2019

chanshunli commented Nov 15, 2019

cnuernber commented Nov 15, 2019

chanshunli commented Nov 15, 2019

chanshunli commented Nov 15, 2019

Thanks @cnuernber Chris, It's work good 👍 👍 👍

cnuernber commented Nov 15, 2019

chanshunli commented Nov 16, 2019

cnuernber commented Nov 16, 2019

cnuernber commented Nov 17, 2019

chanshunli commented Nov 18, 2019

cnuernber commented Nov 18, 2019

support python `with` keyword #8

support python `with` keyword #8