Skip to content

Elasticsearch init ops and kernel#1098

Merged
yongtang merged 5 commits into
tensorflow:masterfrom
kvignesh1420:es-init
Aug 24, 2020
Merged

Elasticsearch init ops and kernel#1098
yongtang merged 5 commits into
tensorflow:masterfrom
kvignesh1420:es-init

Conversation

@kvignesh1420

@kvignesh1420 kvignesh1420 commented Aug 21, 2020

Copy link
Copy Markdown
Member

This PR addresses the issue #1093 by adding implementations for the following:

  • IO>ElasticsearchReadableInit Op.
  • ElasticsearchReadableInitOp Kernel.
  • tfio.experimental.elasticsearch.ElasticsearchIODataset template implementation and few helper methods.

Also, modified the BUILD file for Bazel build of elasticsearch_ops.

@kvignesh1420

Copy link
Copy Markdown
Member Author

@yongtang the build on windows fails due to :

elasticsearch_ops.lo.lib(elasticsearch_kernels.obj) : error LNK2019: unresolved external symbol "public: __cdecl
 tensorflow::CurlHttpRequest::CurlHttpRequest(void)" (??0CurlHttpRequest@tensorflow@@QEAA@XZ) referenced in 
function "public: virtual class tensorflow::HttpRequest * __cdecl tensorflow::CurlHttpRequest::Factory::Create(void)" (?
Create@Factory@CurlHttpRequest@tensorflow@@UEAAPEAVHttpRequest@3@XZ)
2020-08-21T11:53:47.8604213Z bazel-out\x64_windows-fastbuild\bin\tensorflow_io\core\python\ops\libtensorflow_io.so : fatal error LNK1120: 1 unresolved externals
2020-08-21T11:53:47.9588587Z Target //tensorflow_io/core:python/ops/libtensorflow_io.so failed to build

It is unable to resolve CurlHttpRequest. Not sure why this is happening as the only other build using this class is core_ops and it seems to pass.

cc_library(
    name = "core_ops",
    srcs = [
        "kernels/archive_kernels.cc",
        "kernels/http_kernels.cc",
        "ops/core_ops.cc",
    ],
    copts = tf_io_copts(),
    linkstatic = True,
    deps = [
        ":dataset_ops",
        "@curl",
        "@libarchive",
    ],
    alwayslink = 1,
)

@yongtang

Copy link
Copy Markdown
Member

@kvignesh1420 that was a known issue and has not been resolved yet. As you might see, core_ops is not enabled for windows build:

    ] + select({
        "@bazel_tools//src/conditions:windows": [],
        "//conditions:default": [
            "//tensorflow_io/core:core_ops",
            "//tensorflow_io/core:genome_ops",
            "//tensorflow_io/core:optimization",
            "//tensorflow_io/core:oss_ops",
            "//tensorflow_io/core/kernels/gsmemcachedfs:gs_memcached_file_system",
        ],
    }),

You can just move "//tensorflow_io/core:elasticsearch_ops", to the line below "//tensorflow_io/core:core_ops", and Windows will skip the elasticsearch op.

@kvignesh1420

kvignesh1420 commented Aug 21, 2020

Copy link
Copy Markdown
Member Author

cc: @yongtang for review.

@yongtang yongtang left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@yongtang yongtang merged commit 8e83b11 into tensorflow:master Aug 24, 2020
@yongtang

Copy link
Copy Markdown
Member

@kvignesh1420 Thanks for the PR! Would you mind to add a test in a follow up PR?

@kvignesh1420 kvignesh1420 deleted the es-init branch August 24, 2020 23:50
@kvignesh1420

Copy link
Copy Markdown
Member Author

@yongtang sure.

@kvignesh1420

kvignesh1420 commented Aug 25, 2020

Copy link
Copy Markdown
Member Author

@yongtang can we return a tensor of tensors from the kernel to the OP?

For ex: The records I get from the api response are a list of jsons. In this scenario, I want to prepare a tensor for the column/field names and a tensor<tensor> where each inner tensor will contain the values for a column/field. The issue is, how do I return this tensor<tensor> to python from cpp kernels. Is there an alternative approach for these scenarios?

@yongtang

Copy link
Copy Markdown
Member

@kvignesh1420 I would assume an easy way is to obtain the json string from dataset implementation, then using dataset.map() internally (could be part of the ElasticsearchIODataset class) to deserialize the json string into a structure of tensors. You can take a look at tfio.experimental.serialization.decode_json and see if this fits the need.

Let me know if you run into any issues.

i-ony pushed a commit to i-ony/io that referenced this pull request Feb 8, 2021
* elasticsearch init ops, kernel and python api

* lint fixes

* added curl dependency

* BUILD lint fixes

* skipping window's build for elasticsearch ops
zheolong pushed a commit to zheolong/io-1 that referenced this pull request Jul 24, 2025
* elasticsearch init ops, kernel and python api

* lint fixes

* added curl dependency

* BUILD lint fixes

* skipping window's build for elasticsearch ops
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants