Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pubber endpoint configuration working with mosquitto broker #497

Merged
merged 23 commits into from
Nov 9, 2022
Merged
Show file tree
Hide file tree
Changes from 18 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 10 additions & 7 deletions .gencode_hash.txt
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,9 @@ b35646a2f3d7c30fd34e3ea6ab6a1070b39c469fd1de49cab1093ff32faaf06d gencode/docs/c
e1f62882576d0f4332360f2aedb79b37a43973fb31d862c829af924f8f310ec1 gencode/docs/command_mapping.html
5513a43ef916d0413de0c95aede03042ce8824e98fbb8f3634c12f5dee53cce8 gencode/docs/config.html
30c7f39e50e290b0a36a3414cf1c90277ebd249f126f85f0a307bc7b74164e08 gencode/docs/config_mapping.html
5598402bde04860394d91ecf43f5c653ff539cce49d011db4b5b17d596b72c2a gencode/docs/configuration_endpoint.html
01beba6a34f69b73897a9b59ff0730de317b76f997cef13f79eec16d555ae14a gencode/docs/configuration_endpoint.html
b7ca70e3176f997f254c32bd8d53dff1462ac8a71c0e166aaa85193d67bff60b gencode/docs/configuration_execution.html
00e8646fd8117d47ddf32f2488228cd35b66fd711cb21288dc6bb5a5715978cb gencode/docs/configuration_pubber.html
260f0f3c7ebd901941b7d007197c8a24e203e785555f153bce00736caea97698 gencode/docs/configuration_pubber.html
0f0355e343cb8197dfbae1481c0eca3a0566d73181bc968e33fd469a1c94d1e6 gencode/docs/envelope.html
f1fde870638541bea6de16acd64365c89a7d56ee3270d6fd1bcdbb5bdb141887 gencode/docs/event.html
a8800ab8384de3b846af0d72cf310acf01644842b557e42eec20b69a0bfa1868 gencode/docs/event_discovery.html
Expand All @@ -13,7 +13,7 @@ a8800ab8384de3b846af0d72cf310acf01644842b557e42eec20b69a0bfa1868 gencode/docs/e
35238ac2be3b2860bf74d291342897ef53a537a86aeb37d0eab55e3c8f8baecb gencode/docs/event_system.html
43025f61381de050eae9680c3193a1d878a397d0b5515dd1063a1a993b0c41ea gencode/docs/event_validation.html
9ae793a641f2e053c5af734bf813cd2637a675330acb8d9e903ef03b4286e007 gencode/docs/metadata.html
e101ec26541b7787fa722244fed744713d05649de547160f39744e1e61850ddd gencode/docs/persistent_device.html
b4d3081ffe9f3473b105f0b47ad27121455165ded83deb5adc8178cc1c6104cc gencode/docs/persistent_device.html
5d039d607af9ec75ee552dfe36b16c702687ea16f5663f41fc49b4533b86e00d gencode/docs/properties.html
392f272c1c8cd047eb6906f2f11ffacc7083c32f639da2437d6a9ca861062ecc gencode/docs/readme.md
690d56e96c4e180e49e0abcba358ab1842ea06dd3f693918852a9de370c994d5 gencode/docs/reflect_config.html
Expand All @@ -25,7 +25,9 @@ aa1fb0f836ccfdcb275a6763771fdc4b3a3b90a4e2e1cca97e4a929e9075c284 gencode/docs/s
776f696f93961b7da9fab8494d4487aea01533a8df9aab7968687f63de2f9f91 gencode/docs/state_mapping.html
06f9e538fbab613144d6d5f64274464850f60f4f730b6a04ad0e049ce34276ec gencode/docs/state_validation.html
d39d7fe37a41c74a40080af7b0a429d201ab1fdff7444428c4b98eb7b38c332b gencode/java/udmi/schema/Asset.java
b405ce628f7819b46b19950aeaba89ee938fea54261000616bc534b9f81bd59c gencode/java/udmi/schema/Auth_provider.java
0825a5cec83003bb0a6488c4ed7010a04ae0d3848ef36fe01bb4e6718ba7b96d gencode/java/udmi/schema/Aux.java
ce2c747fab0d374987acc51474a52ca5b3d64659d51cffa671d5442b7114339a gencode/java/udmi/schema/Basic.java
2812fb9daa2c66ae3b145e903978e3dab718ee8ed5439cfa9424409dcb8f179f gencode/java/udmi/schema/BlobBlobsetConfig.java
a7c57d119adcd0cf6363cc5301ce562004222522242e8ffd1d0cd7010f235ae1 gencode/java/udmi/schema/BlobBlobsetState.java
b9f903444ab08907e41eb123286434ff3207b1edd01397af3ddefb8475bbdadc gencode/java/udmi/schema/BlobsetConfig.java
Expand All @@ -43,7 +45,7 @@ d8a80ab3180d33bfa17564c969018e1d4350a47dbc70c4ae8a5abbfb25cfedc9 gencode/java/u
4793b2370b1f3b8fd89941586b779dffdde83d321a2621907f67d4d297d19b17 gencode/java/udmi/schema/DiscoveryEvent.java
04112dd47b0f761131c276c67d3cd8b789d25e6716b5732be9fef14fc6831f1d gencode/java/udmi/schema/DiscoveryModel.java
9962b0eb7d5adf52af6160e9f3131f8eeb52ae9e518954dbb6aead1bcad0245e gencode/java/udmi/schema/DiscoveryState.java
b90ab40a281c5e383881629bd8a791ba3d30491e06ebc623a476e2735a3f6c3a gencode/java/udmi/schema/EndpointConfiguration.java
5fe6d4b0605f09b8bd954646556e604b1867610755c133bf6288c835dd48ed88 gencode/java/udmi/schema/EndpointConfiguration.java
6fea7710d4eb64bfc568820f7063a89ce7808853ef7a56763fa90f323e09362f gencode/java/udmi/schema/Entry.java
017f8e237efa959b81d72f3dd2e78b915856198ceef02c041c657b30df93b7c2 gencode/java/udmi/schema/Envelope.java
e9f5c77be81486b6b8c6d88f70f2d50583d8c3fafa2ac09ead80f44b8d5e751e gencode/java/udmi/schema/Event.java
Expand All @@ -56,6 +58,7 @@ efb376cd7a80ca74fbf74d3ef3f4101d393341a91d01db36456aa84f6be3d0a5 gencode/java/u
60a8115ae1acae7c199b63180823198d38ec50d57b48dd85aca1ccc865058f85 gencode/java/udmi/schema/GatewayConfig.java
56b46f4914ef1f4baa59bf597186ff7901b7c8b607720ec798f4e4e6ad59aa08 gencode/java/udmi/schema/GatewayModel.java
e0e7739046e834c0f0ca6a70b38b4579618899be3162887a0fa7ab60bbff22a5 gencode/java/udmi/schema/GatewayState.java
b0d4bff14a65ebddc2dd253c996708a4cce99592e82978f057f32a6d9c7768a1 gencode/java/udmi/schema/Jwt.java
a5e5adfc187709e8646a11c92e804acfb67743f9d72149008aaca954df3177f6 gencode/java/udmi/schema/Level.java
07fd4911363437b274c19b024759b04b116152176702da8d4203c4ff4cb55b7f gencode/java/udmi/schema/LocalnetConfig.java
910c68183db7703b00bcb81146ad73e6fe0d4bbc4caec4dc9c621f3cc2e5eee5 gencode/java/udmi/schema/LocalnetModel.java
Expand All @@ -79,7 +82,7 @@ fc3a9415c04d8a06954dbdbfdff5d68ab113cce3948532c19df555778ffb04fa gencode/java/u
61afd6c9c0364b8982c90df7664147829bad8858408b2141299ee8bbc35f1612 gencode/java/udmi/schema/PointsetSummary.java
ca2e7566106818ca7e5190c8041eb86f0c9b3251b0bda8c3ea7ce11a0c891a0a gencode/java/udmi/schema/Position.java
3df66bb1a37a9e0b2b6cf392f8c64d404a73c83e5e13c02bb4844f09b9a04b70 gencode/java/udmi/schema/Properties.java
faa1cd11467f9e58742cfee7124d4af04dbf481eb5a80549bbd2eb77bcc8059d gencode/java/udmi/schema/PubberConfiguration.java
783320700f7e96cf8e421461f14999afca10cd540ca6aadf184add39f0ac048c gencode/java/udmi/schema/PubberConfiguration.java
a5737c70324cf13322c019049f80e334bf699dea2f434859762fe3553fee8308 gencode/java/udmi/schema/PubberOptions.java
7bc53cb9a61df8f804a75aa4860403d19281f7c665cbddd5364a2aa746a1f651 gencode/java/udmi/schema/ReflectorConfig.java
abe99dd74122c186403baa6982300a9d5968f8bbb7a67b1689104111b98f32fb gencode/java/udmi/schema/ReflectorState.java
Expand Down Expand Up @@ -119,9 +122,9 @@ ac3facbd96f7cb2f7e387e7497d6a36af379a2687329571f250c5670f9933244 gencode/python
607c5047df878a1333df3ce88dcce34668959b0b315f6954bf1a4963dcf7839e gencode/python/udmi/schema/config_pointset_point.py
ba37d2d54df565aba42ef055a0ec961175d180c2e8092a914e62029bc5388857 gencode/python/udmi/schema/config_system.py
97c2e5fadc6da0d84660f3296de885ab59f4b04154179b6717f77ec366f1544b gencode/python/udmi/schema/config_system_testing.py
4637fc257ea4d6ea5a46d98db69c57e27e4ecbf5d86f77036cd1382f1a40f532 gencode/python/udmi/schema/configuration_endpoint.py
f1a865b9953f4549f3bbe1cf6543e31be9d4db640bc82031c520fe5fffe65b26 gencode/python/udmi/schema/configuration_endpoint.py
e8a1574074554b6144b178d2adedc76a1c7be5ae911b253deff4460d8d82c6ee gencode/python/udmi/schema/configuration_execution.py
3814c88403934dbd3fce77d92a8ad45c68dc7e07c319ba9b4e8f1b1ac7518c07 gencode/python/udmi/schema/configuration_pubber.py
7f4f8c1a6ead34bf72b9134b97a39001752eb0f9647fafaac82ced4bb2f46b88 gencode/python/udmi/schema/configuration_pubber.py
998ce105f88686f27b85f3630a396ed04b106f830c133a684ea5c505ca95b1c3 gencode/python/udmi/schema/envelope.py
1eb9019b9d0b4ff7de2df8beb625a4f89292d636ece0c02f160495c537bd6c2c gencode/python/udmi/schema/event.py
82182e3f569ad80dc0751027959c7db9135d10072fbe79f896d63a4cd2f4771f gencode/python/udmi/schema/event_discovery.py
Expand Down
7 changes: 5 additions & 2 deletions .github/workflows/testing.yml
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,6 @@ jobs:
redirect:
name: Endpoint Redirection
runs-on: ubuntu-latest
needs: udmi # Access to UDMI-REFLECTOR is mutually exclusive
timeout-minutes: 10
steps:
- uses: actions/checkout@v2
Expand All @@ -56,7 +55,10 @@ jobs:
run: |
bin/setup_base
bin/clone_model
- name: pubber runs
- name: mosquitto connect
run: |
bin/test_mosquitto
- name: pubber redirect
env:
GCP_TARGET_PROJECT: ${{ secrets.GCP_TARGET_PROJECT }}
if: "${{ env.GCP_TARGET_PROJECT != '' }}"
Expand Down Expand Up @@ -95,6 +97,7 @@ jobs:
udmi:
name: Integration Tests
runs-on: ubuntu-latest
needs: redirect # Access to UDMI-REFLECTOR is mutually exclusive
timeout-minutes: 20
steps:
- uses: actions/checkout@v2
Expand Down
1 change: 1 addition & 0 deletions bin/pubber
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,7 @@ cat <<EOF > /tmp/pubber_config.json
},
"sitePath": "$site_path",
"deviceId": "$device_id",
"projectId": "$project_id",
"serialNo": "$serial_no",
"options": $options_json
}
Expand Down
2 changes: 1 addition & 1 deletion bin/setup_base
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
#!/bin/bash

sudo apt-get install -y hxtools moreutils expect
sudo apt-get install -y hxtools moreutils expect mosquitto mosquitto-clients

python3 --version
python3 -m venv venv
Expand Down
80 changes: 80 additions & 0 deletions bin/test_mosquitto
Original file line number Diff line number Diff line change
@@ -0,0 +1,80 @@
#!/bin/bash -e

ROOT_DIR=$(dirname $0)/..
cd $ROOT_DIR

if [[ $# != 0 ]]; then
echo Usage: $0
false
fi

site_path=sites/udmi_site_model
device_id=AHU-1 # Static device for testing. Needs to be different than other tests scripts so as not to conflict during CI.
serial_no=$RANDOM

site_config=$site_path/cloud_iot_config.json
cloud_region=$(jq -r .cloud_region $site_config)
registry_id=$(jq -r .registry_id $site_config)

mkdir -p out
PASSWORD_FILE=/etc/mosquitto/test_mosquitto.passwd
USERNAME=scrumptus
PASSWORD=aardvark

echo password_file ${PASSWORD_FILE} | sudo tee /etc/mosquitto/conf.d/udmi_test.conf
sudo sed -i 's/allow_anonymous true/allow_anonymous false/' /etc/mosquitto/mosquitto.conf
sudo sed -i 's/#listener/listener/' /etc/mosquitto/mosquitto.conf

sudo touch ${PASSWORD_FILE}
sudo mosquitto_passwd -b ${PASSWORD_FILE} ${USERNAME} ${PASSWORD}
sudo chmod 666 ${PASSWORD_FILE}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Eventually we can't do this kind of stuff

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Which kind of stuff, exactly?


sudo systemctl restart mosquitto
systemctl status mosquitto
killall mosquitto_sub || true

echo Subscribing to mqtt/test in the background...
mosquitto_sub -t \# -u ${USERNAME} -P ${PASSWORD} | tee out/mosquitto.sub &

sleep 1

echo Publishing test message
mosquitto_pub -t mqtt/test -u ${USERNAME} -P ${PASSWORD} -m "Hello MQTT"

sleep 1

cat <<EOF > /tmp/pubber_config.json
{
"endpoint": {
"protocol": "mqtt",
"transport": "tcp",
"client_id": "$registry_id/$device_id",
"hostname": "127.0.0.1",
"port": 1883,
"auth_provider": {
"basic": {
"username": "$USERNAME",
"password": "$PASSWORD"
}
}
},
"sitePath": "$site_path",
"deviceId": "$device_id",
"serialNo": "$serial_no"
}
EOF

ps ax | fgrep java | fgrep /tmp/pubber_config.json | fgrep -v grep | awk '{print $1}' | xargs -r kill
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

something like I'm doing eleswhere?

function kill_existing() {
kill $(pgrep -f 'java.*pubber-.*jar')
}

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done.


$ROOT_DIR/pubber/bin/build

timeout 10s $ROOT_DIR/pubber/bin/run /tmp/pubber_config.json || true

received_no=$(fgrep operational out/mosquitto.sub | jq -r .system.serial_no)

if [[ $received_no != $serial_no ]]; then
echo Mismatched/missing serial no: $received_no != $serial_no
false
fi

echo Done with mosquitto pubber check.
7 changes: 6 additions & 1 deletion bin/test_redirect
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,12 @@ cat <<EOF > out/endpoint.json
{
"protocol": "mqtt",
"client_id": "projects/$project_id/locations/$cloud_region/registries/reconfigure/devices/$device_id",
"hostname": "mqtt.googleapis.com"
"hostname": "mqtt.googleapis.com",
"auth_provider": {
"jwt": {
"audience": "$project_id"
}
}
}
EOF

Expand Down
Loading