vineyard v0.2.7

Vineyard v0.2.7 brings many new features into experimental stage, including

• Airflow Integration with Vineyard: vineyard serves as a XCom backend for airflow to support the tasks in user-defined workflow shares large volume of intermediate data using vineyard, and enables more possibilities for orchestrating bigdata analytical workloads with Airflow. To try Airflow with vineyard, please refer to https://v6d.io/notes/airflow.html

• Vineyard brings Dask integration in this release, data in vineyard can be consumed and produced by dask procedures, and can be further shared by other compute engines that works on Vineyard. For a end-to-end demo, please refer to https://v6d.io/notes/dask.html

• Vineyard now can serve as the data source for ML frameworks, e.g., tensorflow, pytorch and xgboost. (thanks @chaitravi-ce )

• Vineyard Go SDK (thanks @linlih) and Java SDK (thanks @zhanglei1949) comes into a very preliminary stage.

Vineyard v0.2.7 will the last release of v0.2.x series and v0.3.0 will come next week!

Commits

• [491fbbc]: Upgrade vineyard to v0.2.7, and add a version number config in headers. (Tao He)
• [cc1d2af]: Fixes a bug in read response and native lib output directory, and adapt to Vineyard Java Fragment. (#453) (Tao He)
• [7050e63]: Fix a tiny bug in tensorflow resolver (#451) (Diwen Zhu)
• [00c5bfe]: Fixes the RPATH problem on Mac. (#450) (Tao He)
• [bbe1c69]: Add doc for a dask tensorflow example (#440) (Diwen Zhu)
• [be6f4ad]: Bump junit from 4.11 to 4.13.1 in /java/modules (#447) (dependabot[bot])
• [1b36571]: Add a vineyard::Blob -> ArrowBuf resolver. (#446) (Tao He)
• [8c6357a]: The objectmeta interface, and packaging script for FFI resolver on metadata (#441) (Tao He)
• [c2ccc65]: Implements a static constructor to get metadata from unsafe sources. (#445) (Tao He)
• [7d99778]: Ensure mmap in the same order with send_fd in server. (#444) (Tao He)
• [84f2164]: Drop unused rust code from the /src directory. (#443) (Tao He)
• [a558ed1]: Catch JSON error as well when reading the RPC request from client. (#442) (Tao He)
• [65482c5]: Memory allocator tests and benchmarks. (#438) (Pei Li)
• [1ec9f19]: Add ipc, proposal and rpc for Rust (#437) (Sijie)
• [2724ca7]: Add doc and quick fix to xgboost (#436) (Diwen Zhu)
• [46d820d]: Fix invalid step (#435) (Siyuan Zhang)
• [5830e1a]: Add CI for Rust (#434) (Siyuan Zhang)
• [d4db525]: Use unique_ptr to allow better flexibility for FFI interfaces. (#433) (Tao He)
• [c964ae3]: Add Go SDK put/get/drop name api implementation (#432) (linlih)
• [aa9bd17]: Add a troubleshooting section about the "permission error". (#431) (Tao He)
• [00278cf]: Added Global Resolvers and Tests for PyTorch and Documentation (#416) (Chaitravi Chalke)
• [3add0e7]: Add docs and example dags for airflow integration. (#429) (Tao He)
• [d34a4ca]: Bug Fix: property_graph_util::VertexDataColumn subtraction between shared_ptr and ptrdiff_t (#428) (Zhang Lei)
• [364cbfa]: Refine the resolver context implementation, and refactor duplicated code (#425) (Tao He)
• [e7ed7c3]: fix template type in vertexDataColum (#424) (Zhang Lei)
• [38d6957]: Lint: reduce code duplication, remove unused imports, isort, etc. (#421) (Tao He)
• [794ce15]: Add a command line arguments to control whether printing the metrics. (#420) (Tao He)
• [793b5df]: Add missing init.py to dask module otherwise it is an empty package. (#417) (Tao He)
• [05820a5]: Adds migrate command to vineyard-ctl (#419) (Rohan Gupta)
• [a5d3c5c]: [MRG]🎉 init prometheus setting (#418) (Jiang Shanshan)
• [06fe9df]: Add Dask integration: vineyard as data sink (#415) (Diwen Zhu)
• [d800c4f]: Airflow: support running among distributed celery workers. (#414) (Tao He)
• [5b16173]: Add Dask integration: vineyard as data source (#409) (Diwen Zhu)
• [262c484]: Added Tests for Tensorflow (#410) (Chaitravi Chalke)
• [22055c2]: Associate the client object to resolved value as well when possible. (#413) (Tao He)
• [dca1b69]: Vineyard XCom backend for apache-airflow. (#411) (Tao He)
• [f729e93]: Upgrade etcd-cpp-apiv3 to make it compatible with cmake less than 3.10. (Tao He)
• [b8efbb6]: Added Global Resolvers for Tensorflow and Local Builders and Resolvers for PyTorch (#395) (Chaitravi Chalke)
• [fb448e3]: Allow the IPC socket connectable by other users. (#407) (Tao He)
• [44bf8f4]: Adds documentation for vineyard-ctl (#405) (Rohan Gupta)
• [b1268cf]: Adds new commands, arguments and examples to vineyard-ctl (#401) (Rohan Gupta)
• [e03cf51]: Run xgboost tests use a scoped fixture, rather than populating the global ctx (#403) (Tao He)
• [b448a9f]: Add docs for the resolver/builder/driver context. (Tao He)
• [5505cc7]: Doc chapter for Vineyard ctl. (#402) (Tao He)
• [dd987b1]: Tidy up contrib/ml/xgboost with new resolver context (#400) (Diwen Zhu)
• [ad03562]: Contextual resolver/builder/driver for flexible nested dynamic registry. (#399) (Tao He)
• [4b4f9f8]: Enable contrib-ml test, and install xgboost deps. (#396) (Tao He)
• [26b40d4]: Add a doc chapter for ML integration. (#397) (Tao He)
• [7e0d44a]: Add xgboost integration and test (#394) (Diwen Zhu)
• [71e326b]: Add go sdk rpc/ipc client connect and close api implementation (#392) (linlih)
• [7f6ec86]: Fixes write bytes and formatting of objectmapper. (#390) (Tao He)
• [[e5978cc](https://github.com/alibaba/libviney...

vineyard v0.2.6

Vineyard: bump version to v0.2.6.

Signed-off-by: Tao He <linzhu.ht@alibaba-inc.com>

vineyard v0.2.5

Vineyard v0.2.5 includes a lot of performance improvements.

Commits

• [789b19f]: Fixes the doc: we don't have vineyard.{array,dataframe}. (#355) (Tao He)
• [84a039d]: Don't do iterate & delete-by-key at the same time. (#354) (Tao He)
• [d921873]: Support unicode path in io adaptor (#353) (Siyuan Zhang)
• [5162953]: Fixes the buffer overflow error in get_demangled_name. (#351) (Tao He)
• [c1dcc64]: Graceful clean-up on bulk store.(#333) (#350) (Pei Li)
• [d739501]: Improve the implementation of create_metadata. (#347) (Tao He)
• [889e198]: Resolve a FIXME, avoid redundant copy when building contents to blob. (#349) (Tao He)
• [490b8f2]: Suggest use make -j$(nproc) to users. (#348) (Tao He)
• [5f6505b]: Enhance the tensor builder to return a customized subclass. (#346) (Tao He)
• [d420518]: Put implementation for platform-specific APIs into a common env.cc. (#344) (Tao He)
• [ebad256]: Initialize the "handled_rev_" with a proper value. (#343) (Tao He)
• [ac4df16]: Fixes the state binding in metadata.items(). (#342) (Tao He)
• [b60ab12]: Revise the install instruction for apache-arrow. (#340) (Tao He)
• [88ee1d1]: Enhance the support for migration+deletion. (#338) (Tao He)
• [e349cbe]: [FEATURE-316] Condition to check maximum shared memory size (#322) (Rohan Gupta)
• [5862692]: Fixs issues in associate a backtrace to erroneous status.(#326) (#335) (Pei Li)
• [ab1ce67]: Install shared-memory38 as deps for python<=3.7 as well. (#334) (Tao He)
• [015a457]: Revise the issue/pull-request template on Github. (#332) (Tao He)
• [b93b342]: Improved Documentation and Added Patch (#324) (Chaitravi Chalke)
• [6e9c9cc]: Use a std::mutex to protect registered_callbacks_. (#331) (Tao He)
• [e6aa468]: Adding timing information when running pytest. (#328) (Tao He)
• [9091175]: Adding the shared-memory38 dependency. (#329) (Tao He)
• [c0ca234]: Improve the efficiency of vineyardd by avoid establishing new watches in requestUpdates (#313) (Diwen Zhu)

vineyard v0.2.4

Vineyard v0.2.4 refactor the CI settings for better development experience, other enhancement and fixes are highlighted as follows:

• Vineyard v0.2.4 adds a SyncMetadata method on the client to issue a sync request immediately to fetch updates from etcd.
• Vineyard v0.2.4 has finished a lot improvements on the Python SDK, especially support for bytes, memoryview and NumPy types.
• An optional command-line arguments -rpc/-norpc is added to vineyardd to conditionally disable the RPC service.
• Vineyard v0.2.4 enhances the start_vineyardd a lot to make launching vineyard server easier for fast testing.
• Vineyard v0.2.4 has better support for mars, see also mars#2162.

Commits

• [1e224ac]: Enhancement on vineyard resolver/builder for better support for native types. (#319) (Tao He)
• [dc42e51]: Fixes issues in deploying vineyardd using python interface. (#320) (Tao He)
• [040e562]: Add a command-line option to vineyardd for disabling the RPC service. (#318) (Tao He)
• [f36224b]: Update MAINTAINERS to add an explicit email column. (Tao He)
• [1d5e1c0]: Fixes the inconsistent description in docs. (#314) (Tao He)
• [28f56a9]: Reduce the CI loads to save time and avoid been disabled. (Tao He)
• [1b616c5]: Object builder for tuple and pair: accept an objectid as well. (#312) (Tao He)
• [be69d5a]: Add a "SyncMetaData" method on client to sync remote metadata in etcd immediately. (#310) (Tao He)
• [3f8988b]: Bump up the etcd-cpp-apiv3 version for a bugfix in keepalive. (Tao He)
• [0d7d888]: Test runner: put etcd data dir to /tmp when running on MacOS. (#308) (Tao He)
• [8a9b573]: Fix incorrect length of offset array (#306) (Siyuan Zhang)
• [4a9fd2b]: Hotfix for #295 (#300) (Siyuan Zhang)
• [3c91911]: Remove mapping generation when validate (#299) (Siyuan Zhang)
• [7fff7d5]: Add summer of code redirection. (Tao He)
• [0b04596]: Add google analytical links to docs. (Tao He)

vineyard v0.2.3

Commits

• [5cfaa6f]: Enhancement and bugfix of fragment related stuffs. (#295) (Siyuan Zhang)
• [98a0b7d]: Fixes data-race in delete (local) metadata, and start heartbeat when previous round finished. (#293) (Tao He)
• [8b6d937]: Implements "client.AllocatedSize" API. (#292) (Tao He)
• [1ef5807]: Enhance the helm charts implementation. (#288) (Tao He)
• [777aad8]: Fixes for 254db77 and 2c3e3f8. (Tao He)
• [254db77]: Fixes copy artifacts in CI script. (Tao He)
• [2c3e3f8]: Fixes vineyardd location in Docker image. (Tao He)
• [daa78ea]: Use vineyard's logo as favicon.ico. (Tao He)
• [8e73f6f]: Fixes etcd image tag, as "latest" doesn't exist anymore. (Tao He)
• [f9b2f8b]: Docutils 0.17.x has problem with sphinx_rtd_theme. (Tao He)
• [b5e906a]: Add CNCF logo in svg format. (Tao He)
• [f0a350a]: Update README.rst (Jingbo Xu)
• [5b5c955]: Refine roadmaps, and we don't requires CLA anymore, rather, we ask for DCO. (Tao He)

vineyard latest: pre-release for latest vineyard

Commits

• 4a9fd2b: Hotfix for #295 (#300) (Siyuan Zhang) #300

vineyard v0.2.2

Vineyard 0.2.2 is the first pre-release after vineyard joined CNCF as a sandbox project. In this release,

• A prototype for the kubernetes scheduler plugin that aware the locality of vineyard objects and jobs that requires those objects has been published.
• The segmentation fault bug in stacktrace has been fixed.
• The etcd-cpp-apiv3 has been updated to the latest release as several concurrent bugs related to "lock" action on etcd has been fixed.

Commits

• [01e1e50]: A helper to provide a bytes interface for r/w on Python object. (#286) (Tao He)
• [90b9e7d]: CI: hotfix for docker-push error. (Tao He)
• [b549f9a]: Implements vineyard-operator to schedule jobs based on data locality. (#223) (Tao He)
• [bf019cf]: Use a customized version of tmate action to make debug easier. (#285) (Tao He)
• [cf1fbf5]: Add tiny logo for CNCF and sandbox. (Tao He)
• [457fc49]: Use sandbox logo in README. (Tao He)
• [3c9133a]: Enable CodeQL test on CI. (#284) (Tao He)
• [5be0d27]: Upgrade etcd-cpp-apiv3, and add CNCF badge in README. (Tao He)
• [ad42ad2]: Fixes codacy errors. (#283) (Tao He)
• [3b9ae8c]: Resize of CNCF footer. (Tao He)
• [11efa26]: FOSSA and CII Best Practices badage. (Tao He)
• [08dc673]: Add CNCF logo into repo. (Tao He)
• [3e36ae6]: Add footer configuration and site meta for SEO. (Tao He)
• [0c28121]: Move to v6d-io, and add CNCF footer. (Tao He)
• [5d2ecf4]: Update vineyard logo. (Tao He)
• [73a5095]: Add more concurrent tests on meta service. (#282) (Tao He)
• [4dd2702]: Put the "License" part to the end of the page. (Tao He)
• [96c06be]: Update the FAQ section in documentation (#281) (Diwen Zhu)
• [3e1b6e0]: Fixes dots. (Tao He)
• [9f0f5ae]: Include a FAQ page in documentation. (Tao He)
• [61904dd]: Force install clang-format. (Tao He)
• [fc74d7c]: Migrate slack to CNCF. (Tao He)
• [5733c01]: Update the arrow web link (#278) (HouliangQi)
• [2415bb1]: Update vineyard logo. (Tao He)
• [f5164f5]: Make sure the vineyard server is gracefully stopped. (#271) (Tao He)
• [64a99b1]: Fixes stacktrace by avoiding use TLS sections. (#269) (Tao He)
• [cd4d222]: Add column to vertex: support both arrow::Array and (aligned)arrow::ChunkedArray. (#268) (Tao He)
• [e902c3c]: Fixes a bug in LocalIOAaptor: the operator & doesn't has specified sequential order. (#267) (Tao He)
• [30d3e14]: Ensure vineyard could be killed using ctrl-c. (#266) (Tao He)
• [27ce702]: Enabling treating vineyard as a submodule for an external project. (Tao He)
• [e77b692]: Keep backwards compatiblity. (Tao He)
• [0d1126c]: Rename EXPERIMENTAL_ON to NETWORKX (#265) (Weibin Zeng)

vineyard v0.2.1

Vineyard 0.2.1 makes the IOAdaptor extensible and new kinds of IO supported can be registered easier, and disables backtrace for gserror as a temporary workaround for a segmentation fault bug.

Commits

• [e07df24]: Disable the stacktrace for gserror, as there's a unknown failure. (Tao He)
• [70c791d]: Leave a space for \0 at the end. (#264) (Tao He)
• [4c82eb5]: Re-enable migration tests on CI. (#263) (Tao He)
• [155e8d0]: Update submodules, and enhance test as well coverage. (#262) (Tao He)
• [653a9a0]: Make io factory works in a register mode to make it extensible. (#261) (Tao He)

vineyard v0.2.0

Commits

• [9eb8060]: Fixes a bug in output_buffer management when print stacktrace. (#260) (Tao He)
• [e4317ea]: Create parent directories when it not exists, and do better for termination. (#259) (Tao He)
• [31ce570]: Optimize use of gcov to improve coverage reporting. (#258) (Tao He)
• [2c37b7e]: Fixes a crucial bug about creating empty blobs. (Tao He)
• [439b7a9]: Makes the ctrl-c safe for every stage during booting. (#256) (Tao He)
• [3644d85]: Rename the repo from libvineyard to v6d. (#255) (Tao He)
• [8ea25a5]: Unify the implementation on IO adaptors to fully leverage arrow. (#254) (Tao He)
• [28f39ae]: Glob files by prefix. (#253) (Siyuan Zhang)
• [6cab155]: Implements the multi-thread async server for vineyardd, with necessary concurrency control. (#230) (Tao He)
• [23d5551]: Add ListArray. (#247) (Siyuan Zhang)
• [dec3573]: Allowing folly::dynamics of object/array/double/bool type to be hashed“ (#246) (Weibin Zeng)
• [3d40be8]: Client-space allocator based on Jemalloc. (#237) (Tao He)

vineyard v0.1.15

Commits

• [5c1a734]: Support merge multiple add vertices and edges operation into one. (#245) (Siyuan Zhang)
• [4ca66a5]: Upgrade etcd-cpp-apiv3 submodule. (Tao He)
• [fc6aade]: Not raise if the process is already ended (#242) (Siyuan Zhang)
• [d179fb3]: Fix unsupported date temporarily (#241) (Siyuan Zhang)
• [7694d66]: Hot fix for #239: the property id is not continuous when used in MaxGraphSchema. (#240) (Siyuan Zhang)