Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Strengthen the schema w/r/t format, required fields #397

Merged
merged 70 commits into from
Jul 28, 2022
Merged
Show file tree
Hide file tree
Changes from 66 commits
Commits
Show all changes
70 commits
Select commit Hold shift + click to select a range
4b3824a
Add dryrun option -n
johnrandolph Jun 2, 2022
e0c6de0
Oops, shift by the option
johnrandolph Jun 2, 2022
a6b72b2
Remove dbg
johnrandolph Jun 2, 2022
bf78213
Merge remote-tracking branch 'faucet/master'
johnrandolph Jun 2, 2022
5e9de32
Merge remote-tracking branch 'faucet/master'
johnrandolph Jun 3, 2022
00ac587
Merge remote-tracking branch 'faucet/master'
johnrandolph Jun 7, 2022
b60ecdb
First pass at RegistrarTest and slight changes to Registrar.
johnrandolph Jun 9, 2022
2ad6e87
Add validation tests
johnrandolph Jun 14, 2022
d159768
Test work
johnrandolph Jun 21, 2022
6793eff
Test improvements
johnrandolph Jun 22, 2022
0f662bb
Merge branch 'master' into master
johnrandolph Jun 22, 2022
167e8de
Merge
johnrandolph Jun 22, 2022
eeb32b4
Fixes from first review
johnrandolph Jun 22, 2022
cc2f8b6
Move execute to method, keep argv parsing in class
johnrandolph Jun 23, 2022
7afa5b0
Import
johnrandolph Jun 23, 2022
bac9875
Merge remote-tracking branch 'faucet/master'
johnrandolph Jun 23, 2022
fdfda2e
Lint fixes
johnrandolph Jun 23, 2022
45ea044
Up the warnings
johnrandolph Jun 23, 2022
645b3fb
Basic Pubber Redirect (#376)
grafnu Jun 27, 2022
ffd812e
Add preliminary base64 payload prep/apply/commit blob msgs.
johnrandolph Jun 28, 2022
a1ae005
Proposal that it may be beneficial to hint the type that base64 decode
johnrandolph Jun 28, 2022
5ddfdc5
Merge remote-tracking branch 'faucet/swarm' into swarm
johnrandolph Jun 28, 2022
65771ad
Model ids as map key name
johnrandolph Jun 28, 2022
cf7731b
Model map key name as id
johnrandolph Jun 28, 2022
0f38b05
Prep
johnrandolph Jun 28, 2022
6144588
Refactor pubber endpoint config (#377)
grafnu Jun 28, 2022
62aab52
Merge remote-tracking branch 'faucet/swarm' into swarm
johnrandolph Jun 29, 2022
520733f
Updates for blobset = map of blobs{id}, phase inside blob
johnrandolph Jun 29, 2022
bf73b63
Add type=url for retrieving http/https resources
johnrandolph Jun 29, 2022
dca05b5
Add two examples of retrieving a zip
johnrandolph Jun 29, 2022
304b829
Add proposed format for a config blob in json
johnrandolph Jun 29, 2022
253dcaa
Change to base64: "inline_payload", url: "url", etc.
johnrandolph Jun 30, 2022
b95edfc
Update examples for apply and commit
johnrandolph Jul 1, 2022
32ab6b5
Update commit.json
johnrandolph Jul 13, 2022
421cb97
Delete prep.json
johnrandolph Jul 13, 2022
8433c33
Delete apply.json
johnrandolph Jul 13, 2022
c348259
Change hash to sha256
johnrandolph Jul 14, 2022
5089cbc
Rename commit to applied
johnrandolph Jul 14, 2022
43e5ce0
Beginning of writing state replies for blobset config msgs.
johnrandolph Jul 14, 2022
de98d66
Add further examples
johnrandolph Jul 18, 2022
f603cb3
Add category, message, remove phase
johnrandolph Jul 19, 2022
cff3e9d
rm
johnrandolph Jul 19, 2022
42dc8ba
Add required fields
johnrandolph Jul 19, 2022
cce99b6
Delete blobset_base64_error.json
johnrandolph Jul 20, 2022
d36c1ab
Delete blobset_receive.json
johnrandolph Jul 20, 2022
7be3f5a
Re-commit
johnrandolph Jul 20, 2022
cf67d64
Revert "Delete blobset_receive.json"
johnrandolph Jul 20, 2022
7a5fafa
Merge branch 'swarm' of github.com:johnrandolph/udmi into swarm
johnrandolph Jul 20, 2022
d4f5793
Delete blobset_url_error.json
johnrandolph Jul 20, 2022
876e9a0
Add it back
johnrandolph Jul 20, 2022
0bc7ee0
Merge branch 'swarm' of github.com:johnrandolph/udmi into swarm
johnrandolph Jul 20, 2022
13eb936
Again
johnrandolph Jul 20, 2022
af5d9b7
Remove plural
johnrandolph Jul 20, 2022
f0bc914
Add schema update
johnrandolph Jul 21, 2022
8f4fc95
Add _ to blobset{ "_key" } regex
johnrandolph Jul 21, 2022
094c0a2
Add
johnrandolph Jul 21, 2022
e8846cc
Add "expected reference file"s
johnrandolph Jul 22, 2022
a01aab3
Add gencode
johnrandolph Jul 22, 2022
1f3e236
Minimal regex
johnrandolph Jul 22, 2022
4a4d1bb
Debug this a bit
johnrandolph Jul 25, 2022
89e2a2f
Remove old test steady
johnrandolph Jul 26, 2022
84939e1
gencode updates
johnrandolph Jul 26, 2022
901c6ed
Merge branch 'master' into blobcfg
johnrandolph Jul 26, 2022
9a40d26
Hash difference?
johnrandolph Jul 26, 2022
984eb16
Merge branch 'blobcfg' of github.com:johnrandolph/udmi into blobcfg
johnrandolph Jul 26, 2022
103fd3f
Merge branch 'master' into blobcfg
johnrandolph Jul 27, 2022
4ec47fe
Make the regex more accurate (a-f) and specific (lowercase)
johnrandolph Jul 28, 2022
0ca90a1
Merge branch 'blobcfg' of github.com:johnrandolph/udmi into blobcfg
johnrandolph Jul 28, 2022
02a9fad
Put this back the way it was for this submit
johnrandolph Jul 28, 2022
5cdfa1e
Updates because of a-z to a-f change
johnrandolph Jul 28, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .gencode_hash.txt
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
83cf5112955016827f7db491348459ac602bbd46ac29bb885dccaf513b5b3df6 gencode/docs/config.html
8c0f20d1da08192edceb6c0c3d3b6de85a2cc5abd5d8188fc56000605871de7b gencode/docs/config.html
e2944b13db5ff06be9caea51d03bca48f2cb093a8bb583dca14051255d34ea6b gencode/docs/envelope.html
04b55d0e207495a1c471423a5e89347342241bbd3c845de3e8355209692eb70a gencode/docs/event_discovery.html
8133e380e40f27c56accbffc665b2eeb56ec84a4da3b52ba7aa5e439c9c40572 gencode/docs/event_pointset.html
Expand All @@ -11,7 +11,7 @@ a82821e72af6d0ee35800e6262eb9bb05256309b98aed2dad1a368fd2d6882bb gencode/docs/i
43019239edfdad71647c40d254b9b3aec68f3deb0c808ba636530c1cf9985fe0 gencode/docs/state.html
d39d7fe37a41c74a40080af7b0a429d201ab1fdff7444428c4b98eb7b38c332b gencode/java/udmi/schema/Asset.java
0825a5cec83003bb0a6488c4ed7010a04ae0d3848ef36fe01bb4e6718ba7b96d gencode/java/udmi/schema/Aux.java
773223fd813648a9fb1eb890287f45fa127ef19a553186395f05a1a47d818716 gencode/java/udmi/schema/BlobBlobsetConfig.java
d5adb804697243f97cdd8589750401654f3fab075a9aeac4f2851e46695ef11d gencode/java/udmi/schema/BlobBlobsetConfig.java
2c03651cb2ecda072b1418222eebb5560185669f8ffdd03021ad5ad8ff7ba3b0 gencode/java/udmi/schema/BlobBlobsetState.java
d2c5b5aae8db27b68104fc83a1f38de0a3f1b5d683f2b13599adf24e96c7d124 gencode/java/udmi/schema/BlobsetConfig.java
fcbed49f1af8b791d8c52bcbe18f65521a79d9ac3eb33ec3afd9b342ab2bfc56 gencode/java/udmi/schema/BlobsetState.java
Expand Down
10 changes: 7 additions & 3 deletions bin/check_version
Original file line number Diff line number Diff line change
Expand Up @@ -15,10 +15,14 @@ if [[ -z $origin ]]; then
origin=faucetsdn
fi

git fetch $origin --tags 2>/dev/null
git fetch $origin --tags # 2>/dev/null
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

unexpected #


upstream_ver=`git describe $origin/$branch --abbrev=0 || echo unknown`
current_ver=`git describe HEAD --abbrev=0 || echo unknown`
if [[ $? -ne 0 ]]; then
current_ver="unknown"
else
upstream_ver=`git describe $origin/$branch --abbrev=0 || echo unknown`
current_ver=`git describe HEAD --abbrev=0 || echo unknown`
fi

if [[ $current_ver != $upstream_ver ]]; then
echo
Expand Down
120 changes: 116 additions & 4 deletions gencode/docs/config.html

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

3 changes: 2 additions & 1 deletion gencode/java/udmi/schema/BlobBlobsetConfig.java

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

15 changes: 11 additions & 4 deletions schema/config_blobset_blob.json
Original file line number Diff line number Diff line change
Expand Up @@ -11,13 +11,20 @@
"type": "string"
},
"base64": {
"type": "string"
"type": "string",
"pattern": "^[0-9A-Za-z=+/]{4,}$"
},
"url": {
"type": "string"
"type": "string",
"format": "uri"
},
"sha256": {
"type": "string"
"type": "string",
"pattern": "^[0-9A-Za-z]{64}$"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what do you think about just making it all-lower or all-upper? Reason being, then it's easier to do a naive "string compare" to see if "hash matched" -- otherwise it takes extra code that somebody is likely to get wrong... I'd rather force the generation to be all-lower than rely on the equality check.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm fine with all lowercase because that's customary. I was wondering if device makers would have other opinions, especially from non-Unix platforms.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also this is a mistake and it should be a-f not a-z.

}
}
},
"oneOf": [
{ "required": ["content_type", "base64"] },
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would have expected oneOf { base64, url } (transport mechanism), and then sha256 is always required, and then content_type is optional? I suppose sha256 isn't required with base64... But doesn't content_type potentially apply to all transport mechanisms? Oh wait, is it because content_type is included in the HTTP response?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, content_type comes from http so we don't want to allow overriding it--feels dangerous to me

{ "required": ["url", "sha256"] }
]
}
23 changes: 23 additions & 0 deletions tests/config.tests/blobset_applied.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
{
"version": 1,
"blobset": {
"blobs": {
"arbitrary_manufacturer_id": {
"phase": "applied",
"content_type": "application/zip",
"base64": "UEsDBAoAAAAAAAZZ3VSU7ZiuBQAAAAUAAAAIABwAYm9vdC5pbWdVVAkAA9xqvGLcarxidXgLAAEEFjQAAARTXwEAYm9vdApQSwMECgAAAAAAB1ndVK9osDMDAAAAAwAAAAYAHABmdy5pbWdVVAkAA95qvGLearxidXgLAAEEFjQAAARTXwEAZncKUEsBAh4DCgAAAAAABlndVJTtmK4FAAAABQAAAAgAGAAAAAAAAQAAAKSBAAAAAGJvb3QuaW1nVVQFAAPcarxidXgLAAEEFjQAAARTXwEAUEsBAh4DCgAAAAAAB1ndVK9osDMDAAAAAwAAAAYAGAAAAAAAAQAAAKSBRwAAAGZ3LmltZ1VUBQAD3mq8YnV4CwABBBY0AAAEU18BAFBLBQYAAAAAAgACAJoAAACKAAAAAAA="
},
"_firmware_update": {
"phase": "applied",
"url": "https://manuf.com/path/file.zip",
"sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
},
"_endpoint_redirect": {
"phase": "applied",
"content_type": "application/json",
"base64": "e30="
}
}
},
"timestamp": "2022-07-13T12:00:00.000Z"
}
File renamed without changes.
12 changes: 12 additions & 0 deletions tests/config.tests/blobset_applied_incomplete_payload.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
{
"version": 1,
"blobset": {
"blobs": {
"arbitrary_manufacturer_id": {
"phase": "applied",
"base64": "UEsDBAoAAAAAAAZZ3VSU7ZiuBQAAAAUAAAAIABwAYm9vdC5pbWdVVAkAA9xqvGLcarxidXgLAAEEFjQAAARTXwEAYm9vdApQSwMECgAAAAAAB1ndVK9osDMDAAAAAwAAAAYAHABmdy5pbWdVVAkAA95qvGLearxidXgLAAEEFjQAAARTXwEAZncKUEsBAh4DCgAAAAAABlndVJTtmK4FAAAABQAAAAgAGAAAAAAAAQAAAKSBAAAAAGJvb3QuaW1nVVQFAAPcarxidXgLAAEEFjQAAARTXwEAUEsBAh4DCgAAAAAAB1ndVK9osDMDAAAAAwAAAAYAGAAAAAAAAQAAAKSBRwAAAGZ3LmltZ1VUBQAD3mq8YnV4CwABBBY0AAAEU18BAFBLBQYAAAAAAgACAJoAAACKAAAAAAA="
}
}
},
"timestamp": "2022-07-13T12:00:00.000Z"
}
2 changes: 2 additions & 0 deletions tests/config.tests/blobset_applied_incomplete_payload.out
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
1 schema violations found
instance failed to match exactly one schema (matched 0 out of 2)
12 changes: 12 additions & 0 deletions tests/config.tests/blobset_applied_incomplete_url.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
{
"version": 1,
"blobset": {
"blobs": {
"_firmware_update": {
"phase": "applied",
"url": "https://manuf.com/path/file.zip"
}
}
},
"timestamp": "2022-07-13T12:00:00.000Z"
}
2 changes: 2 additions & 0 deletions tests/config.tests/blobset_applied_incomplete_url.out
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
1 schema violations found
instance failed to match exactly one schema (matched 0 out of 2)
12 changes: 0 additions & 12 deletions tests/config.tests/steady.json

This file was deleted.

Empty file removed tests/config.tests/steady.out
Empty file.