Added support for loading Python at runtime (supports both Python 3 and 2) #20674

pvieito · 2018-11-18T20:57:59Z

Added new PythonLibrary struct to load Python library and symbols dynamically at runtime

This adds support for both Python 3 and 2 with the same codebase.
You can force a version of Python with the PYTHON_VERSION environment variable, a Python library path with PYTHON_LIBRARY and enable logging with PYTHON_LOADER_LOGGING.
The implementation comes from the PythonKit Swift package (https://github.com/pvieito/PythonKit) removing the Foundation dependency.

xwu · 2018-11-19T01:46:54Z

stdlib/public/Python/PythonLibrary.swift

+    }
+
+    for majorVersion in supportedMajorVersions {
+      for minorVersion in supportedMinorVersions {


Would this be more elegant if there were simply an array of tuples called supportedVersions: [(major: Int, minor: Int?)]?

Such a tuple could also be manually populated (i.e., [(3, nil), (3, 8), (3, 7), /* ... */]). There's no need for supporting nonexistent Python versions (indeed, that would be an oxymoron).

Removing support for "nonexistent" Python versions means that old Swift distributions (or even any distributed executable that statically linked the Python module) would not work with any 3.x Python version released after it, even though the Python ABI is marked as stable for all the Python 3 lifetime, so it should work. Also, it would require maintaining the list and deciding when it's time to add new versions.

Take in account that while in Windows the Python distribution includes both python3.dll and python37.dll in macOS and Linux we have to specify the exact minor version to load it (Python.framework/Versions/3.7/Python or libpython3.7m.so).

If the ABI is stable for all versions of Python 3, then is there a need for a list of supported minor versions?

@xwu Read the second paragraph.

I'm afraid I don't understand. Why do we need a list of supported minor versions? If a specific version isn't specified, you'd want to load the most recent version of Python 3 regardless of what minor version it is, no?

@xwu When we dynamically load Python with dlopen we need to know the filename of the library, but as each minor version has a different version we have to try each one until one loads. For example in macOS:

$ PYTHON_LOADER_LOGGING=1 PYTHON_VERSION=3 PythonTool Loading symbol 'Py_Initialize' from the Python library... Trying to load library at 'Python.framework/Versions/3/Python'... Trying to load library at '/usr/local/Frameworks/Python.framework/Versions/3/Python'... Trying to load library at 'Python.framework/Versions/3.30/Python'... Trying to load library at '/usr/local/Frameworks/Python.framework/Versions/3.30/Python'... Trying to load library at 'Python.framework/Versions/3.29/Python'... Trying to load library at '/usr/local/Frameworks/Python.framework/Versions/3.29/Python'... Trying to load library at 'Python.framework/Versions/3.28/Python'... Trying to load library at '/usr/local/Frameworks/Python.framework/Versions/3.28/Python'... Trying to load library at 'Python.framework/Versions/3.27/Python'... Trying to load library at '/usr/local/Frameworks/Python.framework/Versions/3.27/Python'... Trying to load library at 'Python.framework/Versions/3.26/Python'... Trying to load library at '/usr/local/Frameworks/Python.framework/Versions/3.26/Python'... Trying to load library at 'Python.framework/Versions/3.25/Python'... Trying to load library at '/usr/local/Frameworks/Python.framework/Versions/3.25/Python'... Trying to load library at 'Python.framework/Versions/3.24/Python'... Trying to load library at '/usr/local/Frameworks/Python.framework/Versions/3.24/Python'... Trying to load library at 'Python.framework/Versions/3.23/Python'... Trying to load library at '/usr/local/Frameworks/Python.framework/Versions/3.23/Python'... Trying to load library at 'Python.framework/Versions/3.22/Python'... Trying to load library at '/usr/local/Frameworks/Python.framework/Versions/3.22/Python'... Trying to load library at 'Python.framework/Versions/3.21/Python'... Trying to load library at '/usr/local/Frameworks/Python.framework/Versions/3.21/Python'... Trying to load library at 'Python.framework/Versions/3.20/Python'... Trying to load library at '/usr/local/Frameworks/Python.framework/Versions/3.20/Python'... Trying to load library at 'Python.framework/Versions/3.19/Python'... Trying to load library at '/usr/local/Frameworks/Python.framework/Versions/3.19/Python'... Trying to load library at 'Python.framework/Versions/3.18/Python'... Trying to load library at '/usr/local/Frameworks/Python.framework/Versions/3.18/Python'... Trying to load library at 'Python.framework/Versions/3.17/Python'... Trying to load library at '/usr/local/Frameworks/Python.framework/Versions/3.17/Python'... Trying to load library at 'Python.framework/Versions/3.16/Python'... Trying to load library at '/usr/local/Frameworks/Python.framework/Versions/3.16/Python'... Trying to load library at 'Python.framework/Versions/3.15/Python'... Trying to load library at '/usr/local/Frameworks/Python.framework/Versions/3.15/Python'... Trying to load library at 'Python.framework/Versions/3.14/Python'... Trying to load library at '/usr/local/Frameworks/Python.framework/Versions/3.14/Python'... Trying to load library at 'Python.framework/Versions/3.13/Python'... Trying to load library at '/usr/local/Frameworks/Python.framework/Versions/3.13/Python'... Trying to load library at 'Python.framework/Versions/3.12/Python'... Trying to load library at '/usr/local/Frameworks/Python.framework/Versions/3.12/Python'... Trying to load library at 'Python.framework/Versions/3.11/Python'... Trying to load library at '/usr/local/Frameworks/Python.framework/Versions/3.11/Python'... Trying to load library at 'Python.framework/Versions/3.10/Python'... Trying to load library at '/usr/local/Frameworks/Python.framework/Versions/3.10/Python'... Trying to load library at 'Python.framework/Versions/3.9/Python'... Trying to load library at '/usr/local/Frameworks/Python.framework/Versions/3.9/Python'... Trying to load library at 'Python.framework/Versions/3.8/Python'... Trying to load library at '/usr/local/Frameworks/Python.framework/Versions/3.8/Python'... Trying to load library at 'Python.framework/Versions/3.7/Python'... Trying to load library at '/usr/local/Frameworks/Python.framework/Versions/3.7/Python'... Library at '/usr/local/Frameworks/Python.framework/Versions/3.7/Python' was sucessfully loaded. Loading symbol 'PyEval_GetBuiltins' from the Python library... Loading symbol 'Py_IncRef' from the Python library... Loading symbol 'PyImport_ImportModule' from the Python library... Loading symbol 'PyObject_GetAttrString' from the Python library... Loading symbol 'Py_DecRef' from the Python library... Loading symbol 'PyLong_AsLong' from the Python library... Loading symbol 'PyErr_Occurred' from the Python library... [*] Python 3.7 [ ] Version: 3.7.1

And on Linux:

$ PYTHON_LOADER_LOGGING=1 PYTHON_VERSION=3 PythonTool Loading symbol 'Py_Initialize' from the Python library... Trying to load library at 'libpython3.so'... Trying to load library at 'libpython3m.so'... Trying to load library at 'libpython3.30.so'... Trying to load library at 'libpython3.30m.so'... Trying to load library at 'libpython3.29.so'... Trying to load library at 'libpython3.29m.so'... Trying to load library at 'libpython3.28.so'... Trying to load library at 'libpython3.28m.so'... Trying to load library at 'libpython3.27.so'... Trying to load library at 'libpython3.27m.so'... Trying to load library at 'libpython3.26.so'... Trying to load library at 'libpython3.26m.so'... Trying to load library at 'libpython3.25.so'... Trying to load library at 'libpython3.25m.so'... Trying to load library at 'libpython3.24.so'... Trying to load library at 'libpython3.24m.so'... Trying to load library at 'libpython3.23.so'... Trying to load library at 'libpython3.23m.so'... Trying to load library at 'libpython3.22.so'... Trying to load library at 'libpython3.22m.so'... Trying to load library at 'libpython3.21.so'... Trying to load library at 'libpython3.21m.so'... Trying to load library at 'libpython3.20.so'... Trying to load library at 'libpython3.20m.so'... Trying to load library at 'libpython3.19.so'... Trying to load library at 'libpython3.19m.so'... Trying to load library at 'libpython3.18.so'... Trying to load library at 'libpython3.18m.so'... Trying to load library at 'libpython3.17.so'... Trying to load library at 'libpython3.17m.so'... Trying to load library at 'libpython3.16.so'... Trying to load library at 'libpython3.16m.so'... Trying to load library at 'libpython3.15.so'... Trying to load library at 'libpython3.15m.so'... Trying to load library at 'libpython3.14.so'... Trying to load library at 'libpython3.14m.so'... Trying to load library at 'libpython3.13.so'... Trying to load library at 'libpython3.13m.so'... Trying to load library at 'libpython3.12.so'... Trying to load library at 'libpython3.12m.so'... Trying to load library at 'libpython3.11.so'... Trying to load library at 'libpython3.11m.so'... Trying to load library at 'libpython3.10.so'... Trying to load library at 'libpython3.10m.so'... Trying to load library at 'libpython3.9.so'... Trying to load library at 'libpython3.9m.so'... Trying to load library at 'libpython3.8.so'... Trying to load library at 'libpython3.8m.so'... Trying to load library at 'libpython3.7.so'... Trying to load library at 'libpython3.7m.so'... Trying to load library at 'libpython3.6.so'... Trying to load library at 'libpython3.6m.so'... Trying to load library at 'libpython3.5.so'... Trying to load library at 'libpython3.5m.so'... Library at 'libpython3.5m.so' was sucessfully loaded. Loading symbol 'PyEval_GetBuiltins' from the Python library... Loading symbol 'Py_IncRef' from the Python library... Loading symbol 'PyImport_ImportModule' from the Python library... Loading symbol 'PyObject_GetAttrString' from the Python library... Loading symbol 'Py_DecRef' from the Python library... Loading symbol 'PyLong_AsLong' from the Python library... Loading symbol 'PyErr_Occurred' from the Python library... [*] Python 3.5 [ ] Version: 3.5.2

As you can see, neither in macOS nor on Linux, the major version is enough to locate a valid library (Python.framework/Versions/3/Python/libpython3.so) so we have to try all possible minor versions.

I understand that. What I don't understand is why we need a static array of "supported" minor versions if support is not contingent on the minor version. How many minor versions you try is arbitrary and an implementation detail of the code that loads the library, having nothing to do with what versions are supported.

So what would you suggest?

static func loadPythonLibrary() -> UnsafeMutableRawPointer? { if let pythonLibraryPath = Environment.library.value { return loadPythonLibrary(at: pythonLibraryPath) } let triedMinorVersions: [Int?] = [nil] + Array(0...30).reversed() for majorVersion in supportedMajorVersions { for minorVersion in triedMinorVersions { for libraryPath in libraryPaths { let version = PythonVersion(major: majorVersion, minor: minorVersion) guard let pythonLibraryHandle = loadPythonLibrary( at: libraryPath, version: version) else { continue } return pythonLibraryHandle } } } return nil }

?

We can work on this in a follow-up.

There will never be a version of Python 2 after 2.7, for instance, and it makes no sense to look for one. If "Python 2.30" existed, we certainly wouldn't want to support it, since what would that even be? Likewise, is Python 2.0 really supported? (In a similar vein, I'm not sure that Python 3.30, if it ever exists, should be supported today; at the current clip of one minor release every year or so, that'd be 20 years in the future, and if that were to happen who knows what that version would look like?) This is all to say that iterating over the same minor versions for every major version is not ideal.

rxwei · 2018-11-19T01:46:57Z

stdlib/public/Python/PythonLibrary.swift

+}
+
+// Methods of `PythonLibrary` required to load the Python library.
+extension PythonLibrary {


Make the extension private and drop ‘private’ from members.

rxwei · 2018-11-19T01:48:01Z

stdlib/public/Python/PythonLibrary.swift

+    return pythonLibraryHandle
+  }
+
+  private static func getPythonLibraryHandle() -> UnsafeMutableRawPointer? {


Avoid using “get” or “set” in function names.

Plus, I think this can be a computed property.

rxwei · 2018-11-19T01:50:04Z

stdlib/public/Python/PythonLibrary.swift

+}
+
+// Methods of `PythonLibrary` used for logging messages.
+extension PythonLibrary {


Is this used in other files? If not, make the extension private.

rxwei · 2018-11-19T01:51:03Z

stdlib/public/Python/PythonLibrary.swift

+}
+
+// Methods of `PythonLibrary` required to read the environment variables.
+extension PythonLibrary {


Should this be private too?

rxwei · 2018-11-19T01:52:24Z

stdlib/public/Python/PythonLibrary.swift

+
+// Methods of `PythonLibrary` used for logging messages.
+extension PythonLibrary {
+  static func log(message: String) {


Argument label “message” doesn’t add any clarity. Why not just log(_:)?

rxwei · 2018-11-19T01:53:27Z

stdlib/public/Python/PythonLibrary.swift

+  }
+
+  static func loadSymbol<T>(
+    name: String, legacyName: String? = nil, signature: T.Type


“signature” is an overloaded word. I’d suggest using “type:”.

rxwei

LGTM. Thanks!

We will also need an API that enables the user to select a Python version. Something like: Python.useVersion(3, 6). It can be added in a separate patch.

rxwei · 2018-11-20T20:05:21Z

stdlib/public/Python/PythonLibrary.swift

+        }
+      }
+    }
+


Remove empty line.

xwu · 2018-11-20T23:29:06Z

stdlib/public/Python/PythonLibrary.swift

+
+// Methods of `PythonLibrary` required to load the Python library.
+private extension PythonLibrary {
+  static let supportedMajorVersions: [Int] = Array(2...3).reversed()


Please just write [3, 2].

pvieito · 2018-11-21T00:12:57Z

The function should be on PythonLibrary as merely using Python already calls Py_Initialize().

Something like this would do the trick:

// Methods of `PythonLibrary` required to set the Python version.
extension PythonLibrary {
  private static func useVersion(_ version: PythonVersion) {
    PythonLibrary.Environment.version.set(version.versionString)
  }
  
  public static func useVersion(_ major: Int, _ minor: Int? = nil) {
    let version = PythonVersion(major: major, minor: minor)
    self.useVersion(version)
  }
}

Example:

import Python

PythonLibrary.useVersion(2) // Should be called before any references to `Python`
let sys = try Python.attemptImport("sys")
print(sys.version)

rxwei · 2018-11-21T00:30:41Z

IMO, using a PythonVersion type is unnecessary complexity and is not intuitive. I don't think we need any extra types to represent the python version, or even separate major from minor from an API perspective. The dlopen-based API is fully dynamic, extra types won't give us more safety than just using a Float. How about starting with a very simple API:

public extension PythonLibrary {
  static func useVersion(_ version: Float) {
    ... // do version tuple conversion here and set environment
  }
}

pvieito · 2018-11-21T00:35:20Z

Take in account that the PythonVersion struct is private and it is also used on loadPythonLibrary(at:version:). Also, Float is a really bad idea: 3.10 vs 3.1.

The public method would be: static func useVersion(_ major: Int, _ minor: Int? = nil):

PythonLibrary.useVersion(2)
// or
PythonLibrary.useVersion(2, 7)

rxwei · 2018-11-21T00:53:07Z

Also, Float is a really bad idea: 3.10 vs 3.1.

Ok, I'm sold.

The public method will be: static func useVersion(_ major: Int, _ minor: Int? = nil).

Adding a single method for now and making PythonVersion private sounds good to me.

pvieito · 2018-11-21T01:03:58Z

Adding a single method for now and making PythonVersion private sounds good to me.

Great, done! 👍🏻

rxwei · 2018-11-21T01:05:35Z

@swift-ci please test tensorflow

rxwei · 2018-11-21T01:05:56Z

@swift-ci please test tensorflow

rxwei · 2018-11-21T01:06:18Z

@swift-ci please test tensorflow

rxwei · 2018-11-21T01:06:34Z

CI is dead. Gotta wait a while.

rxwei · 2018-11-21T01:28:23Z

@swift-ci please test tensorflow

rxwei · 2018-11-21T02:13:48Z

Could you add new files to CMakeLists.txt?

pvieito · 2018-11-21T19:51:46Z

@rxwei Should I also remove the CPython module?

dan-zheng · 2018-11-21T20:03:06Z

@pvieito Yes, please remove CPython. I see you've already done so. :)

dan-zheng · 2018-11-21T20:04:27Z

@swift-ci Please test tensorflow

pvieito · 2018-11-21T20:22:17Z

@dan-zheng could you restart the test, please?

I have fixed this error:

/home/swiftci/jenkins/workspace/swift-PR-TensorFlow-Linux/swift/stdlib/public/Python/PythonLibrary.swift:118:19:
note: force-unwrap using '!' to abort execution if the optional value contains 'nil'
      setenv(key, value, 1)
                  ^

But the ones referring to @convention(c) as an unknown expression syntax exists in the source are a bit strange (I have just successfully built PythonKit on Linux with the exact same sources).

dan-zheng · 2018-11-21T20:31:30Z

I believe the unknown expression syntax exists in the source errors are related to libSyntax verification, which is triggered by the flags -Xfrontend -verify-syntax-tree.

Could you try building PythonKit on Linux with those extra flags? Unfortunately, this seems like a libSyntax deficiency. TypeNodes.py may not yet have support for function conventions.

EDIT: It seems that libSyntax does not yet support @convention in all contexts.

// This passes syntax verification.
let f: @convention(c) (Int) -> Int = { x in x }

func foo<T>(_ x: T.Type) {}
// This does not.
foo((@convention(c) (Int) -> Int).self)

$ swiftc -Xfrontend -verify-syntax-tree convention.swift
convention.swift:6:6: error: unknown expression syntax exists in the source
foo((@convention(c) (Int) -> Int).self)
     ^

I filed SR-9316 to track this.

pvieito · 2018-11-21T20:43:53Z

@dan-zheng Exactly:

$ swift build -Xswiftc -Xfrontend -Xswiftc -verify-syntax-tree   
Compile Swift Module 'PythonKit' (4 sources)
Compile Swift Module 'LoggerKit' (12 sources)
/Developer/PythonKit/PythonKit/PythonLibrary+Symbols.swift:39:10: error: unknown expression syntax exists in the source
  type: (@convention(c) () -> ()).self
         ^
/Developer/PythonKit/PythonKit/PythonLibrary+Symbols.swift:44:10: error: unknown expression syntax exists in the source
  type: (@convention(c) (PyObjectPointer?) -> ()).self
         ^
/Developer/PythonKit/PythonKit/PythonLibrary+Symbols.swift:49:10: error: unknown expression syntax exists in the source
  type: (@convention(c) (PyObjectPointer?) -> ()).self
         ^
...

dan-zheng · 2018-11-21T20:58:39Z

@pvieito Hmm, that's unfortunate. Unless we can disable -verify-syntax-tree for the Python module, this PR is blocked by libSyntax support for this usage of @convention.

@nkcsgexi: Could you please provide some advice on how to add libSyntax support for this usage of @convention? (Basically, how to resolve SR-9316). That would be greatly appreciated!

rxwei · 2018-11-21T22:51:39Z

The libSyntax issue is a bit surprising. There's no instance of getting the metatype of a @convention(c) function in stdlib. But let's not get blocked on this. You can work around this by first making type a default argument:

static func loadSymbol<T>(
  name: String, legacyName: String? = nil, type: T.Type = T.self
) -> T

Then change your call sites to provide a contextual type.

let PyList_New: @convention(c) (Int) -> PyObjectPointer? =
  PythonLibrary.loadSymbol(name: "PyList_New")

rxwei · 2018-11-22T00:06:50Z

@swift-ci please test tensorflow

rxwei · 2018-11-22T00:07:06Z

@swift-ci please test tensorflow

rxwei · 2018-11-22T01:17:58Z

@swift-ci please clean test tensorflow

rxwei · 2018-11-22T02:02:58Z

It's still failing possibly because of incorrectly configured cmake. You mentioned you couldn't test it on your machine, is that right? If so, I can take it from here.

pvieito · 2018-11-22T14:28:42Z

You mentioned you couldn't test it on your machine, is that right?

Yes, exactly, and I am not very familiar with cmake.

If so, I can take it from here.

Great, thanks!

pvieito · 2018-12-03T20:15:15Z

It seems that we have to reference the symbol:

guard isType(pythonObject, type: PySlice_Type) else { return nil }

Not its address:

guard isType(pythonObject, type: &PySlice_Type) else { return nil } // Crashes

rxwei · 2018-12-03T20:22:41Z

Indeed! We didn't encounter a test case that would make isType(pythonObject, type: &PyBool_Type) crash.

rxwei · 2018-12-03T20:25:27Z

~~@pvieito could you add me as a collaborator on your swift fork so that I can push to it?~~

Nvm. Pushed.

…ent to `isType(_:type:)`.

rxwei · 2018-12-03T20:28:19Z

@swift-ci please test tensorflow

pvieito · 2018-12-03T20:28:20Z

@rxwei Sure! I suppose we should modify the symbol definition (to match the real Python C header) instead of the calling site?

rxwei · 2018-12-03T20:31:59Z

I suppose we should modify the symbol definition (to match the real Python C header) instead the calling site?

Symbol definitions are correct. It's just that we don't need these type variables to be vars anymore since they are addresses to the actual objects.

pvieito · 2018-12-03T20:33:04Z

Ok, great!

- `()` -> `Void`. - Drop parentheses at the result position from single-result function signatures. - Replace !(==) with != in `isType(_:type:)`.

pvieito · 2018-12-03T20:45:55Z

@rxwei Please, remove also the symbol _Py_TrueStructb (seems to be a copy-paste typo).

rxwei · 2018-12-04T00:33:51Z

@swift-ci please test tensorflow macOS

rxwei · 2018-12-04T04:06:27Z

@swift-ci please test tensorflow macOS

rxwei · 2018-12-04T04:06:33Z

@swift-ci please test tensorflow macOS

rxwei · 2018-12-04T05:34:32Z

@swift-ci please test tensorflow macOS

rxwei · 2018-12-04T05:34:43Z

@swift-ci please test tensorflow macOS

H-World · 2019-12-04T06:11:09Z

@pvieito 我想知道在Mac上怎么使用指定版本的python，现在我使用PythonLibrary.useVersion(3)，会崩溃Python library not found. Set the PYTHON_LIBRARY environment variable with the path to a Python library.但是我又找不到方法去指定PYTHON_LIBRARY

pvieito mentioned this pull request Nov 18, 2018

Added support for loading Python at runtime (supports both Python 3 and 2) #20669

Closed

xwu reviewed Nov 19, 2018

View reviewed changes

rxwei reviewed Nov 19, 2018

View reviewed changes

rxwei requested a review from dan-zheng November 19, 2018 21:11

rxwei approved these changes Nov 20, 2018

View reviewed changes

stdlib/public/Python/PythonLibrary.swift Outdated

}

}

}

Copy link

Contributor

rxwei Nov 20, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove empty line.

xwu reviewed Nov 20, 2018

View reviewed changes

pvieito and others added 2 commits December 3, 2018 12:00

Redefined Python symbols to avoid libSyntax bug (SR-9316)

e1d5249

Update tests

a0e79f3

init(owning:) should retain, while init(borrowing:) should not.

2076fb9

Use loaded symbol values instead of their addresses as the type argum…

d9a2b80

…ent to `isType(_:type:)`.

rxwei added 2 commits December 3, 2018 12:35

Make loaded symbols lets since they are themselves the address we need

9d3d4e0

Minor clean-up.

6267b75

- `()` -> `Void`. - Drop parentheses at the result position from single-result function signatures. - Replace !(==) with != in `isType(_:type:)`.

rxwei added 2 commits December 3, 2018 13:38

Delete _Py_TrueStructb

25290a4

Fix ParsableInterface tests

8f749f4

rxwei merged commit 2214a08 into swiftlang:tensorflow Dec 4, 2018

dan-zheng mentioned this pull request Dec 6, 2018

Add Python 3 support. #18645

Closed

rxwei deleted the python-dynamic-load branch December 6, 2018 13:29

pschuh mentioned this pull request Dec 11, 2018

Bugfix to allow importing numpy again. #21197

Merged

dan-zheng mentioned this pull request Mar 4, 2019

[Python] fix UInt.pythonObject for Python 3.7 #23055

Merged

dan-zheng mentioned this pull request Aug 14, 2020

Fatal error: 'try!' expression unexpectedly raised an error: Python exception: No module named ipykernel.zmqshell google/swift-jupyter#110

Closed

This was referenced Feb 4, 2019

[SR-8726] Use virtual environments in the implementation of python interoperability #51237

Closed

[SR-9316] libSyntax: @convention function type is not supported in all contexts #51786

Closed

Added support for loading Python at runtime (supports both Python 3 and 2) #20674

Added support for loading Python at runtime (supports both Python 3 and 2) #20674

Conversation

pvieito commented Nov 18, 2018

Choose a reason for hiding this comment

pvieito Nov 20, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

xwu Nov 20, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

xwu Nov 20, 2018 • edited Loading

Choose a reason for hiding this comment

pvieito Nov 21, 2018 • edited Loading

Choose a reason for hiding this comment

pvieito Nov 21, 2018 • edited Loading

Choose a reason for hiding this comment

xwu Nov 22, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rxwei left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pvieito commented Nov 21, 2018 • edited Loading

rxwei commented Nov 21, 2018 • edited Loading

pvieito commented Nov 21, 2018 • edited Loading

rxwei commented Nov 21, 2018

pvieito commented Nov 21, 2018

rxwei commented Nov 21, 2018

rxwei commented Nov 21, 2018

rxwei commented Nov 21, 2018

rxwei commented Nov 21, 2018

rxwei commented Nov 21, 2018

rxwei commented Nov 21, 2018

pvieito commented Nov 21, 2018

dan-zheng commented Nov 21, 2018

dan-zheng commented Nov 21, 2018

pvieito commented Nov 21, 2018 • edited Loading

dan-zheng commented Nov 21, 2018 • edited Loading

pvieito commented Nov 21, 2018 • edited Loading

dan-zheng commented Nov 21, 2018

rxwei commented Nov 21, 2018

rxwei commented Nov 22, 2018

rxwei commented Nov 22, 2018

rxwei commented Nov 22, 2018

rxwei commented Nov 22, 2018

pvieito commented Nov 22, 2018

pvieito commented Dec 3, 2018 • edited Loading

rxwei commented Dec 3, 2018 • edited Loading

rxwei commented Dec 3, 2018 • edited Loading

rxwei commented Dec 3, 2018

pvieito commented Dec 3, 2018 • edited Loading

rxwei commented Dec 3, 2018

pvieito commented Dec 3, 2018

pvieito commented Dec 3, 2018

rxwei commented Dec 4, 2018

rxwei commented Dec 4, 2018

rxwei commented Dec 4, 2018

rxwei commented Dec 4, 2018

rxwei commented Dec 4, 2018

H-World commented Dec 4, 2019

pvieito Nov 20, 2018 •

edited

Loading

xwu Nov 20, 2018 •

edited

Loading

xwu Nov 20, 2018 •

edited

Loading

pvieito Nov 21, 2018 •

edited

Loading

pvieito Nov 21, 2018 •

edited

Loading

xwu Nov 22, 2018 •

edited

Loading

pvieito commented Nov 21, 2018 •

edited

Loading

rxwei commented Nov 21, 2018 •

edited

Loading

pvieito commented Nov 21, 2018 •

edited

Loading

pvieito commented Nov 21, 2018 •

edited

Loading

dan-zheng commented Nov 21, 2018 •

edited

Loading

pvieito commented Nov 21, 2018 •

edited

Loading

pvieito commented Dec 3, 2018 •

edited

Loading

rxwei commented Dec 3, 2018 •

edited

Loading

rxwei commented Dec 3, 2018 •

edited

Loading

pvieito commented Dec 3, 2018 •

edited

Loading