Skip to content

Update batch logic to only store+hash the manifest #582

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 14 commits into from
Mar 11, 2022
Merged

Update batch logic to only store+hash the manifest #582

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
1727abd
Initial data structures for batch manifest storage only change
peterbroadhurst Mar 3, 2022
b57f45d
Updates to structures
peterbroadhurst Mar 4, 2022
df20f1b
Merge with main
peterbroadhurst Mar 4, 2022
b4a2956
Update DB interaface to store manifest
peterbroadhurst Mar 4, 2022
fe67f91
Interim work on batch flush state interface
peterbroadhurst Mar 4, 2022
2b903a4
Broadcast interface with batch.DispatchState
peterbroadhurst Mar 4, 2022
9f55e22
Move over the private batch dispatcher
peterbroadhurst Mar 4, 2022
fa48b6b
Implement message cache
peterbroadhurst Mar 7, 2022
6661b22
Move resolve data interface to return full data, so we can populate c…
peterbroadhurst Mar 7, 2022
095c068
Apply changes to batch structure to events
peterbroadhurst Mar 8, 2022
d9844c5
Changes worked through all modules
peterbroadhurst Mar 8, 2022
41a85dc
Merge branch 'main' of github.com:hyperledger/firefly into batch-upgrade
peterbroadhurst Mar 8, 2022
e46b921
Message caching worked through e2e
peterbroadhurst Mar 8, 2022
8952a4f
Close out review comments and stop persisting manifest in input
peterbroadhurst Mar 11, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions db/migrations/postgres/000071_rename_batch_payload_column.down.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
BEGIN;
ALTER TABLE batches RENAME COLUMN manifest TO payload;
COMMIT;
3 changes: 3 additions & 0 deletions db/migrations/postgres/000071_rename_batch_payload_column.up.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
BEGIN;
ALTER TABLE batches RENAME COLUMN payload TO manifest;
COMMIT;
1 change: 1 addition & 0 deletions db/migrations/sqlite/000071_rename_batch_payload_column.down.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
ALTER TABLE batches RENAME COLUMN manifest TO payload;
1 change: 1 addition & 0 deletions db/migrations/sqlite/000071_rename_batch_payload_column.up.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
ALTER TABLE batches RENAME COLUMN payload TO manifest;
334 changes: 128 additions & 206 deletions docs/swagger/swagger.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -903,121 +903,29 @@ paths:
properties:
author:
type: string
blobs:
items: {}
type: array
confirmed: {}
created: {}
hash: {}
id: {}
key:
type: string
manifest:
type: string
namespace:
type: string
node: {}
payload:
properties:
data:
items:
properties:
blob:
properties:
hash: {}
name:
type: string
public:
type: string
size:
format: int64
type: integer
type: object
created: {}
datatype:
properties:
name:
type: string
version:
type: string
type: object
hash: {}
id: {}
namespace:
type: string
validator:
type: string
value:
type: string
type: object
type: array
messages:
items:
properties:
batch: {}
confirmed: {}
data:
items:
properties:
hash: {}
id: {}
type: object
type: array
hash: {}
header:
properties:
author:
type: string
cid: {}
created: {}
datahash: {}
group: {}
id: {}
key:
type: string
namespace:
type: string
tag:
type: string
topics:
items:
type: string
type: array
txtype:
type: string
type:
enum:
- definition
- broadcast
- private
- groupinit
- transfer_broadcast
- transfer_private
type: string
type: object
pins:
items:
type: string
type: array
state:
enum:
- staged
- ready
- sent
- pending
- confirmed
- rejected
type: string
type: object
type: array
tx:
properties:
id: {}
type:
type: string
type: object
type: object
payloadRef:
type: string
tx:
properties:
id: {}
type:
type: string
type: object
type:
enum:
- broadcast
- private
type: string
type: object
description: Success
Expand Down Expand Up @@ -1056,121 +964,29 @@ paths:
properties:
author:
type: string
blobs:
items: {}
type: array
confirmed: {}
created: {}
hash: {}
id: {}
key:
type: string
manifest:
type: string
namespace:
type: string
node: {}
payload:
properties:
data:
items:
properties:
blob:
properties:
hash: {}
name:
type: string
public:
type: string
size:
format: int64
type: integer
type: object
created: {}
datatype:
properties:
name:
type: string
version:
type: string
type: object
hash: {}
id: {}
namespace:
type: string
validator:
type: string
value:
type: string
type: object
type: array
messages:
items:
properties:
batch: {}
confirmed: {}
data:
items:
properties:
hash: {}
id: {}
type: object
type: array
hash: {}
header:
properties:
author:
type: string
cid: {}
created: {}
datahash: {}
group: {}
id: {}
key:
type: string
namespace:
type: string
tag:
type: string
topics:
items:
type: string
type: array
txtype:
type: string
type:
enum:
- definition
- broadcast
- private
- groupinit
- transfer_broadcast
- transfer_private
type: string
type: object
pins:
items:
type: string
type: array
state:
enum:
- staged
- ready
- sent
- pending
- confirmed
- rejected
type: string
type: object
type: array
tx:
properties:
id: {}
type:
type: string
type: object
type: object
payloadRef:
type: string
tx:
properties:
id: {}
type:
type: string
type: object
type:
enum:
- broadcast
- private
type: string
type: object
description: Success
Expand Down Expand Up @@ -10063,5 +9879,111 @@ paths:
description: Success
default:
description: ""
/status/pins:
get:
description: 'TODO: Description'
operationId: getStatusPins
parameters:
- description: Server-side request timeout (millseconds, or set a custom suffix
like 10s)
in: header
name: Request-Timeout
schema:
default: 120s
type: string
- description: 'Data filter field. Prefixes supported: > >= < <= @ ^ ! !@ !^'
in: query
name: batch
schema:
type: string
- description: 'Data filter field. Prefixes supported: > >= < <= @ ^ ! !@ !^'
in: query
name: created
schema:
type: string
- description: 'Data filter field. Prefixes supported: > >= < <= @ ^ ! !@ !^'
in: query
name: dispatched
schema:
type: string
- description: 'Data filter field. Prefixes supported: > >= < <= @ ^ ! !@ !^'
in: query
name: hash
schema:
type: string
- description: 'Data filter field. Prefixes supported: > >= < <= @ ^ ! !@ !^'
in: query
name: index
schema:
type: string
- description: 'Data filter field. Prefixes supported: > >= < <= @ ^ ! !@ !^'
in: query
name: masked
schema:
type: string
- description: 'Data filter field. Prefixes supported: > >= < <= @ ^ ! !@ !^'
in: query
name: sequence
schema:
type: string
- description: Sort field. For multi-field sort use comma separated values (or
multiple query values) with '-' prefix for descending
in: query
name: sort
schema:
type: string
- description: Ascending sort order (overrides all fields in a multi-field sort)
in: query
name: ascending
schema:
type: string
- description: Descending sort order (overrides all fields in a multi-field
sort)
in: query
name: descending
schema:
type: string
- description: 'The number of records to skip (max: 1,000). Unsuitable for bulk
operations'
in: query
name: skip
schema:
type: string
- description: 'The maximum number of records to return (max: 1,000)'
in: query
name: limit
schema:
example: "25"
type: string
- description: Return a total count as well as items (adds extra database processing)
in: query
name: count
schema:
type: string
responses:
"200":
content:
application/json:
schema:
properties:
batch: {}
created: {}
dispatched:
type: boolean
hash: {}
index:
format: int64
type: integer
masked:
type: boolean
sequence:
format: int64
type: integer
signer:
type: string
type: object
description: Success
default:
description: ""
servers:
- url: http://localhost:5000
Loading