Bulk Model Management

[davemoore-]

Implement an API endpoint to submit multiple entity model management operations in bulk, borrowing the functionality for bulk operations introduced in #50. This will enable a more efficient handling of multiple entity model management operations. One envisioned implementation is an entity model management user interface that provides checkboxes to delete multiple models in one request.

Proposed syntax

POST /_zentity/models/_bulk[?PARAMS]
{ PARAMS }
{ PAYLOAD }
...

POST /_zentity/models/ENTITY_TYPE/_bulk[?PARAMS]
{ PARAMS }
{ PAYLOAD }
...

Accepted operations

The bulk endpoint would support operations that create, update, or delete entity models. Unlike the implementation in #50, this will require a field in PARAMS that indicates which action is to be executed. See the Elasticsearch Bulk API implementation for reference, which requires this convention, too.

[davemoore-]

PR #82 pending review.

I'm not implementing the second half of the proposal (see below).

POST /_zentity/models/ENTITY_TYPE/_bulk[?PARAMS]
{ PARAMS }
{ PAYLOAD }
...

It occurred to me that there isn't a compelling use case for multiple bulk operations on a single entity type. Two creates would cause a conflict, two updates would be redundant because each one requires the full document, a create followed by an update would be redundant because updates create the document if it doesn't exist, and two deletes would be redundant.

[austince]

Makes a lot of sense to not have bulk endpoints on a single entity_type scope. I'm sorry I didn't comment earlier, but I think it's a bit odd that the nested PARAMS structure is different than the bulk resolution endpoints. Why not encoding the action as a separate param? I think it might follow the same thinking that people will never be doing multiple operations on a single entity type.

POST /_zentity/models/ENTITY_TYPE/_bulk[?PARAMS]
{ action: "create", "entity_type": "entity_a" }
{ PAYLOAD }
{ action: "delete", "entity_type": "entity_b" }
{ PAYLOAD }

[davemoore-]

My rationale was to keep the syntax consistent with the Elasticsearch Bulk API, which uses the command ("index", "create", "update", "delete") as a top-level field in both the request and response.

But this is a good time to consider the what the syntax really should be, while it's still pre-release. Let's think.

There are remaining differences between the zentity Bulk APIs and the Elasticsearch Bulk API:

• Elasticsearch requires a trailing newline (\n). zentity requires no trailing newline, and to me it feels odd to require it.
• Elasticsearch expects no payload after a "delete" operation. zentity expects pairs of lines for all operations, and to me having inconsistent requirements could be confusing.

Since there are differences, and since Elasticsearch client libraries hide the details of the Bulk API, then to me it raises the question, How similar do the APIs really need to be?

What you propose looks good. I like the consistency.

Going further, what if the concept of pairs was omitted altogether, and replaced with a syntax where one line is one request, containing the params and the body? Examples for both Bulk APIs below:

Bulk Model - Request

POST _zentity/models/_bulk
{ action: ..., entity_type: ..., body: ENTITY_MODEL }
{ action: ..., entity_type: ..., body: ENTITY_MODEL }
{ action: ..., entity_type: ..., body: ENTITY_MODEL }
...

Bulk Model - Response - Same approach as Bulk Resolution: Each response item is simply the response of an individual operation. Don't nested the item under an "action" field.

Bulk Resolution - Request

POST _zentity/resolution/_bulk
{ entity_type: ..., body: RESOLUTION_PAYLOAD }
{ entity_type: ..., body: RESOLUTION_PAYLOAD }
{ entity_type: ..., body: RESOLUTION_PAYLOAD }
...

POST _zentity/resolution/{entity_type}/_bulk
{ body: RESOLUTION_PAYLOAD }
{ body: RESOLUTION_PAYLOAD }
{ body: RESOLUTION_PAYLOAD }
...

Bulk Resolution - Response - No change from current implementation.

What is your reaction on this implementation?

[davemoore-]

My own thinking on the one-line-per-request implementation above is:

• It may be more efficient to process and easier for the client implement.
• It may be more difficult to read the raw request if that's important.
• Mixing params and body on one line might be confusing, or it might not. Would be good to get your opinion.
• It avoids requiring a line with an empty body for "delete" operations, if we believe that would be confusing.

[austince]

Hmm, after your nice examples and reading more about the ES bulk API, I'm actually thinking keeping what is currently implemented is best. I would be interested to know how the ES bulk API responds if multiple actions are given.

While I think the one-line-per-request is slightly easier to understand and use, I don't think it's worth such a big deviation from ES conventions over. Requiring a body after delete makes sense to me though, and I think that's minor enough that we should keep it different.

The most important things, imho, for the bulk formats is that they are supported by current ES tools like Kibana and other clients. The JS client has easy support for encoding bulk formats into NDJSON, so introducing a secondary "bulk" format might only complicate user implementations. I'm not sure, in that case, if it would be easier for the client to implement. I also think separating the params from the body does make the relationship between the bulk and non-bulk formats easier to see.

What are you leaning towards?