Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update arrow to latest master #2100

Merged
merged 17 commits into from May 24, 2018

Conversation

@pcmoritz
Copy link
Contributor

commented May 19, 2018

This includes the following changes:

Serialization fix for Python 2 integers: apache/arrow#2055
Plasma performance improvements: apache/arrow#2056, apache/arrow#2015, apache/arrow#2022
PyTorch 0.4 Tensor serialization fix: apache/arrow#2033

@pcmoritz pcmoritz referenced this pull request May 19, 2018
@AmplabJenkins

This comment has been minimized.

Copy link
Collaborator

commented May 19, 2018

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5492/
Test PASSed.

@AmplabJenkins

This comment has been minimized.

Copy link
Collaborator

commented May 19, 2018

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5504/
Test PASSed.

@robertnishihara

This comment has been minimized.

Copy link
Contributor

commented May 19, 2018

Looks like there are compilation errors on Mac

[ 44%] �[32mBuilding CXX object src/ray/CMakeFiles/ray_objlib.dir/object_manager/object_directory.cc.o�[0m

�[1m/Users/travis/build/robertnishihara/ray-private-travis/src/ray/object_manager/object_store_notification_manager.cc:19:60: �[0m�[0;1;31merror: �[0m�[1m'kDeprecatedPlasmaDefaultReleaseDelay' is deprecated: PLASMA_DEFAULT_RELEASE_DELAY is deprecated [-Werror,-Wdeprecated-declarations]�[0m

      store_client_.Connect(store_socket_name.c_str(), "", PLASMA_DEFAULT_RELEASE_DELAY));

�[0;1;32m                                                           ^

�[0m�[1m/Users/travis/build/robertnishihara/ray-private-travis/thirdparty/pkg/arrow/cpp/build/cpp-install/include/plasma/client.h:39:46: �[0m�[0;1;30mnote: �[0mexpanded from macro 'PLASMA_DEFAULT_RELEASE_DELAY'�[0m

#define PLASMA_DEFAULT_RELEASE_DELAY plasma::kDeprecatedPlasmaDefaultReleaseDelay

�[0;1;32m                                             ^

�[0m�[1m/Users/travis/build/robertnishihara/ray-private-travis/thirdparty/pkg/arrow/cpp/build/cpp-install/include/plasma/client.h:38:19: �[0m�[0;1;30mnote: �[0m'kDeprecatedPlasmaDefaultReleaseDelay' has been explicitly marked deprecated here�[0m

constexpr int64_t kDeprecatedPlasmaDefaultReleaseDelay = 64;

�[0;1;32m                  ^

�[0m1 error generated.
@robertnishihara

This comment has been minimized.

Copy link
Contributor

commented May 19, 2018

Also, this looks related (it happened a couple times on private travis).

======================================================================
FAIL: testLocalSchedulerFailed (__main__.ComponentFailureTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test/component_failures_test.py", line 200, in testLocalSchedulerFailed
    True)
  File "test/component_failures_test.py", line 182, in check_components_alive
    self.assertTrue(component.poll() is None)
AssertionError: False is not true

----------------------------------------------------------------------
Ran 9 tests in 62.892s

FAILED (failures=1)
pcmoritz added 2 commits May 20, 2018
@AmplabJenkins

This comment has been minimized.

Copy link
Collaborator

commented May 20, 2018

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5519/
Test FAILed.

@AmplabJenkins

This comment has been minimized.

Copy link
Collaborator

commented May 20, 2018

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5518/
Test PASSed.

pcmoritz added 2 commits May 20, 2018
fix
@robertnishihara

This comment has been minimized.

Copy link
Contributor

commented May 20, 2018

There's still a linting error

ObjectStoreNotificationManager::ObjectStoreNotificationManager(
     boost::asio::io_service &io_service, const std::string &store_socket_name)
     : store_client_(), socket_(io_service) {
-  ARROW_CHECK_OK(
-      store_client_.Connect(store_socket_name.c_str(), "", plasma::kPlasmaDefaultReleaseDelay));
+  ARROW_CHECK_OK(store_client_.Connect(store_socket_name.c_str(), "",
+                                       plasma::kPlasmaDefaultReleaseDelay));
@AmplabJenkins

This comment has been minimized.

Copy link
Collaborator

commented May 20, 2018

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5522/
Test PASSed.

@AmplabJenkins

This comment has been minimized.

Copy link
Collaborator

commented May 21, 2018

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5528/
Test PASSed.

@robertnishihara

This comment has been minimized.

Copy link
Contributor

commented May 21, 2018

Unfortunately this is still happening on Linux

======================================================================
FAIL: testLocalSchedulerFailed (__main__.ComponentFailureTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test/component_failures_test.py", line 200, in testLocalSchedulerFailed
    True)
  File "test/component_failures_test.py", line 182, in check_components_alive
    self.assertTrue(component.poll() is None)
AssertionError: False is not true

----------------------------------------------------------------------
Ran 9 tests in 62.430s

FAILED (failures=1)
@pcmoritz

This comment has been minimized.

Copy link
Contributor Author

commented May 22, 2018

The test failure is caused by this segfault I think:

Program received signal SIGSEGV, Segmentation fault.
std::unique_ptr<unsigned char [], std::default_delete<unsigned char []> >::unique_ptr(std::unique_ptr<unsigned char [], std::default_delete<unsigned char []> >&&) (
    __u=<optimized out>, this=0x800000000) at /usr/include/c++/5/bits/unique_ptr.h:464
464	      : _M_t(__u.release(), std::forward<deleter_type>(__u.get_deleter())) { }
(gdb) bt
#0  std::unique_ptr<unsigned char [], std::default_delete<unsigned char []> >::unique_ptr(std::unique_ptr<unsigned char [], std::default_delete<unsigned char []> >&&) (
    __u=<optimized out>, this=0x800000000) at /usr/include/c++/5/bits/unique_ptr.h:464
#1  __gnu_cxx::new_allocator<std::unique_ptr<unsigned char [], std::default_delete<unsigned char []> > >::construct<std::unique_ptr<unsigned char [], std::default_delete<unsigned char []> >, std::unique_ptr<unsigned char [], std::default_delete<unsigned char []> > >(std::unique_ptr<unsigned char [], std::default_delete<unsigned char []> >*, std::unique_ptr<unsigned char [], std::default_delete<unsigned char []> >&&) (__p=0x800000000, this=0x2636cc0) at /usr/include/c++/5/ext/new_allocator.h:120
#2  std::allocator_traits<std::allocator<std::unique_ptr<unsigned char [], std::default_delete<unsigned char []> > > >::construct<std::unique_ptr<unsigned char [], std::default_delete<unsigned char []> >, std::unique_ptr<unsigned char [], std::default_delete<unsigned char []> > >(std::allocator<std::unique_ptr<unsigned char [], std::default_delete<unsigned char []> > >&, std::unique_ptr<unsigned char [], std::default_delete<unsigned char []> >*, std::unique_ptr<unsigned char [], std::default_delete<unsigned char []> >&&) (
    __p=0x800000000, __a=...) at /usr/include/c++/5/bits/alloc_traits.h:530
#3  std::deque<std::unique_ptr<unsigned char [], std::default_delete<unsigned char []> >, std::allocator<std::unique_ptr<unsigned char [], std::default_delete<unsigned char []> > > >::emplace_back<std::unique_ptr<unsigned char [], std::default_delete<unsigned char []> > >(std::unique_ptr<unsigned char [], std::default_delete<unsigned char []> >&&) (
    this=0x2636cc0) at /usr/include/c++/5/bits/deque.tcc:155
#4  plasma::PlasmaStore::push_notification (this=this@entry=0x2631d60, object_info=object_info@entry=0x2638f78)
    at /home/ubuntu/ray/thirdparty/build/arrow/cpp/src/plasma/store.cc:633
#5  0x000000000041124f in plasma::PlasmaStore::seal_object (this=this@entry=0x2631d60, object_id=..., 
    digest=digest@entry=0x7ffc5d1d39a0 "\001\253,\271cT", <incomplete sequence \352>) at /home/ubuntu/ray/thirdparty/build/arrow/cpp/src/plasma/store.cc:439
#6  0x0000000000411a34 in plasma::PlasmaStore::process_message (this=0x2631d60, client=<optimized out>) at /home/ubuntu/ray/thirdparty/build/arrow/cpp/src/plasma/store.cc:727
#7  0x0000000000413979 in plasma::PlasmaStore::<lambda(int)>::operator() (events=<optimized out>, __closure=<optimized out>)
    at /home/ubuntu/ray/thirdparty/build/arrow/cpp/src/plasma/store.cc:524
#8  std::_Function_handler<void(int), plasma::PlasmaStore::connect_client(int)::<lambda(int)> >::_M_invoke(const std::_Any_data &, <unknown type in /home/ubuntu/ray/python/ray/core/src/plasma/plasma_store, CU 0x0, DIE 0x82032>) (__functor=..., __args#0=<optimized out>) at /usr/include/c++/5/functional:1871
#9  0x00000000004181da in std::function<void (int)>::operator()(int) const (__args#0=1, this=<optimized out>) at /usr/include/c++/5/functional:2267
#10 plasma::EventLoop::FileEventCallback (loop=<optimized out>, fd=<optimized out>, context=<optimized out>, events=<optimized out>)
    at /home/ubuntu/ray/thirdparty/build/arrow/cpp/src/plasma/events.cc:28
#11 0x0000000000433eb0 in aeProcessEvents (eventLoop=eventLoop@entry=0x2624cc0, flags=flags@entry=3)
    at /home/ubuntu/ray/thirdparty/build/arrow/cpp/src/plasma/thirdparty/ae/ae.c:412
#12 0x00000000004342cb in aeMain (eventLoop=0x2624cc0) at /home/ubuntu/ray/thirdparty/build/arrow/cpp/src/plasma/thirdparty/ae/ae.c:455
#13 0x0000000000418388 in plasma::EventLoop::Start (this=<optimized out>) at /home/ubuntu/ray/thirdparty/build/arrow/cpp/src/plasma/events.cc:69
#14 0x000000000040d2a5 in plasma::PlasmaStoreRunner::Start (use_one_memory_mapped_file=false, hugepages_enabled=false, directory="P;\035]", system_memory=<optimized out>, 
    socket_name=0x7ffc5d1d51d4 "/tmp/plasma_store66283217", this=0x2624c20) at /home/ubuntu/ray/thirdparty/build/arrow/cpp/src/plasma/store.cc:785
#15 plasma::start_server (socket_name=socket_name@entry=0x7ffc5d1d51d4 "/tmp/plasma_store66283217", system_memory=<optimized out>, plasma_directory="/dev/shm", 
    hugepages_enabled=hugepages_enabled@entry=false, use_one_memory_mapped_file=use_one_memory_mapped_file@entry=false)
    at /home/ubuntu/ray/thirdparty/build/arrow/cpp/src/plasma/store.cc:820

pcmoritz added 2 commits May 22, 2018
fix
@AmplabJenkins

This comment has been minimized.

Copy link
Collaborator

commented May 22, 2018

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5559/
Test FAILed.

@AmplabJenkins

This comment has been minimized.

Copy link
Collaborator

commented May 22, 2018

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5557/
Test PASSed.

pcmoritz added 4 commits May 22, 2018
@AmplabJenkins

This comment has been minimized.

Copy link
Collaborator

commented May 22, 2018

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5564/
Test FAILed.

@AmplabJenkins

This comment has been minimized.

Copy link
Collaborator

commented May 22, 2018

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5563/
Test FAILed.

@AmplabJenkins

This comment has been minimized.

Copy link
Collaborator

commented May 22, 2018

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5561/
Test PASSed.

@AmplabJenkins

This comment has been minimized.

Copy link
Collaborator

commented May 22, 2018

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5562/
Test PASSed.

fix
@AmplabJenkins

This comment has been minimized.

Copy link
Collaborator

commented May 22, 2018

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5567/
Test PASSed.

@AmplabJenkins

This comment has been minimized.

Copy link
Collaborator

commented May 22, 2018

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5570/
Test FAILed.

pcmoritz added 2 commits May 23, 2018
fix
fix
@AmplabJenkins

This comment has been minimized.

Copy link
Collaborator

commented May 23, 2018

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5586/
Test PASSed.

@AmplabJenkins

This comment has been minimized.

Copy link
Collaborator

commented May 23, 2018

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5588/
Test FAILed.

@AmplabJenkins

This comment has been minimized.

Copy link
Collaborator

commented May 23, 2018

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5589/
Test FAILed.

@pcmoritz

This comment has been minimized.

Copy link
Contributor Author

commented May 23, 2018

Jenkins retest this please

@AmplabJenkins

This comment has been minimized.

Copy link
Collaborator

commented May 24, 2018

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5605/
Test PASSed.

@richardliaw

This comment has been minimized.

Copy link
Member

commented May 24, 2018

@pcmoritz tests look like they're passing?

@robertnishihara robertnishihara merged commit 225608e into ray-project:master May 24, 2018
1 of 2 checks passed
1 of 2 checks passed
continuous-integration/travis-ci/pr The Travis CI build failed
Details
default Merged build finished.
Details
@robertnishihara robertnishihara deleted the pcmoritz:update-arrow branch May 24, 2018
alok added a commit to alok/ray that referenced this pull request May 25, 2018
* master:
  Prototype named actors. (ray-project#2129)
  Update arrow to latest master (ray-project#2100)
  [DataFrame] Speed up dtypes (ray-project#2118)
  do not fetch from dead Plasma Manager (ray-project#2116)
  [DataFrame] Refactor GroupBy Methods and Implement Reindex (ray-project#2101)
  Initial Support for Airspeed Velocity (ray-project#2113)
  Use automatic memory management in Redis modules. (ray-project#1797)
  [DataFrame] Test bugfixes (ray-project#2111)
  [DataFrame] Update initializations of IndexMetadata which use outdated APIs (ray-project#2103)
alok added a commit to alok/ray that referenced this pull request May 30, 2018
* fix-a3c-torch:
  Prototype named actors. (ray-project#2129)
  Update arrow to latest master (ray-project#2100)
  [DataFrame] Speed up dtypes (ray-project#2118)
  do not fetch from dead Plasma Manager (ray-project#2116)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
4 participants
You can’t perform that action at this time.