Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

allow per-doc metadata to be specified at runtime #358

Closed
costin opened this issue Jan 16, 2015 · 5 comments
Closed

allow per-doc metadata to be specified at runtime #358

costin opened this issue Jan 16, 2015 · 5 comments
Labels
feature :Spark v2.1.0.Beta4

Comments

@costin
Copy link
Member

costin commented Jan 16, 2015

Libraries with API-driven support (like Spark) could support passing the metadata as a pair beside the document. This facilitates computational environments where things like id or the timestamp are created on the fly and are not part of the doc itself.
@costin
Copy link
Member Author

costin commented Jan 19, 2015

Related to #255

@mgreene
Copy link

mgreene commented Jan 23, 2015

@costin Will this include the _version metadata as well?

Also, I'm assuming this issue is talking about the reading of documents? Specifically, when iterating over an RDD of JavaPairRDD<String, Map<String, Object>>, will the Map<String,Object> contain the meta data?

costin added a commit that referenced this issue Jan 23, 2015
@costin
Introduce stand-alone, per-doc, metadata support
1e995ad 
Useful for real-time API environments, such as Spark that can
compute/specify the metadata separately from the document itself

relates #358
relates #255
costin added a commit that referenced this issue Jan 23, 2015
@costin
[DOC] document runtime metadata option for Spark
575a688 
relates #358
relates #255
costin added a commit that referenced this issue Jan 23, 2015
@costin
Polish tests
c56d8f0 
relates #255
relates #358
@costin
Copy link
Member Author

costin commented Apr 28, 2015

Fixed in master. Closing the issue.

@costin costin closed this as completed Apr 28, 2015
@costin costin added the feature label Apr 28, 2015
@mgreene
Copy link

mgreene commented Jun 18, 2015

@costin I don't know if you saw my comment from above but I was wondering how to fetch the metadata while reading documents?

@costin
Copy link
Member Author

costin commented Jun 18, 2015

@mgreene For questions, please use the forum. The issue is about writing (not reading) documents (see the reference commits).
Thanks,

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
feature :Spark v2.1.0.Beta4
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@costin @mgreene