Big rails, ruby and gem version update

[timcowlishaw]

Tried to update as much as possible without breakage / huge changes - this will still need a good test before deploying, but looks good so far.

upgrade rails to 6.1.7.3
Update doorkeeper to version ~> 5
Upgrade Ruby to 3.0.6
upgrade sidekiq to 6.0

ON DEPLOYMENT:

ensure that RAILS_LOG_TO_STDOUT=true is present in the .env file

[timcowlishaw]

@oscgonfer perdona, he arreglado los tests - ahora está listo para revisar!

[oscgonfer]

Hey, some feedback after testing this:

After doing some normal unauthenticated requests, I did two with Auth, to test doorkeeper upgrade. Something seems to be affected by this upgrade as after the first one being successful, the second one gives:

~ curl -X GET 'https://staging-api.smartcitizen.me/v0/devices/1'
An unhandled lowlevel error occurred. The application logs may have details.

And the logs show:

smartcitizen-api-app-1  | source=rack-timeout id=64b3a875-8029-4486-be8a-8a8802050059 timeout=20000ms state=ready
smartcitizen-api-app-1  | source=rack-timeout id=64b3a875-8029-4486-be8a-8a8802050059 timeout=20000ms service=9ms state=active
smartcitizen-api-app-1  | source=rack-timeout id=64b3a875-8029-4486-be8a-8a8802050059 timeout=20000ms service=10ms state=completed
smartcitizen-api-app-1  | 2023-06-12 08:42:14 +0000 Rack app ("GET /v0/devices/1" - (149.34.0.72, 172.70.243.169)): #<TypeError: incompatible marshal file format (can't be read)
smartcitizen-api-app-1  | 	format version 4.8 required; 116.114 given>

[timcowlishaw]

Aha, thanks for testing! I'll take a look ASAP and get to the bottom of it! :-)

[oscgonfer]

Commenting on the sidekiq side of the upgrade, it seems that tasks are being triggered correctly. I have tested with success the DeviceArchive (and delete) workflow, but I would suggest @timcowlishaw that you check the CSV one if possible, similarly to what was done during the tests for #234

[oscgonfer]

Also, adding here - there are these entries in the log:

smartcitizen-api-app-1  | Unsupported HTTP method used: CONNECT
smartcitizen-api-app-1  | 2023-06-13 23:20:46 +0000 HTTP parse error, malformed request: #<Puma::HttpParserError: Invalid HTTP format, parsing fails. Are you trying to open an SSL connection to a non-SSL Puma?>
smartcitizen-api-app-1  | 2023-06-13 23:20:46 +0000 HTTP parse error, malformed request: #<Puma::HttpParserError: Invalid HTTP format, parsing fails. Are you trying to open an SSL connection to a non-SSL Puma?>
smartcitizen-api-app-1  | 2023-06-13 23:20:46 +0000 HTTP parse error, malformed request: #<Puma::HttpParserError: Invalid HTTP format, parsing fails. Are you trying to open an SSL connection to a non-SSL Puma?>
smartcitizen-api-app-1  | 2023-06-13 23:34:40 +0000 HTTP parse error, malformed request: #<Puma::HttpParserError: Invalid HTTP format, parsing fails. Are you trying to open an SSL connection to a non-SSL Puma?>

[timcowlishaw]

Hey @oscgonfer , I have fixed the weird 'marshal format' error (it was, as expected, to do with the throttling), and tested that the CSV job runs successfully - it does, with the caveat that the email doesn't get set (due to a 404 from ActiveDispatch when attempting to contact the mailgun server) - however, I think this is a misconfiguration on staging in general rather than an upgrade problem - the same thing happens when I test from the current master branch. If you can confirm this, then I think I'm happy that this is ready!

[oscgonfer]

Hi! I just tested 1000 requests with unauthorized (no bearer) and I think this last fix would have broken the throttling, see below for logs:

{"id":1,"uuid":"57fec898-fa09-445c-bc30-5eab0c627a3b","name":"Test","description":null,"state":"has_published","postprocessing":null,"hardware_info":{"id":"8517F6DB5055344E312E3120FF0E1943","mac":"BE:FF:4D:07:25:96","time":"2023-06-11T11:14:59Z","esp_bd":"2021-08-26T16:39:05Z","hw_ver":"2.1","sam_bd":"2022-11-22T16:49:09Z","esp_ver":"0.9.8-e283e58","sam_ver":"0.9.8-739fd4a"},"system_tags":["offline"],"user_tags":[],"is_private":false,"notify_low_battery":false,"notify_stopped_publishing":false,"last_reading_at":"2023-06-12T02:57:08Z","added_at":"2022-12-12T11:56:35Z","updated_at":"2023-06-11T11:15:06Z","mac_address":"[FILTERED]","owner":{"id":1,"uuid":"4505db42-12ba-410c-bc53-89a3ea5b3e92","username":"oscar","avatar":"https://smartcitizen.s3.amazonaws.com/avatars/default.svg","url":null,"joined_at":"2022-12-09T17:58:21Z","location":{"city":null,"country":null,"country_code":null},"device_ids":[1,2]},"data":{"recorded_at":"2023-06-12T02:57:08Z","added_at":"2023-06-12T02:57:08Z","location":{"ip":null,"exposure":null,"elevation":null,"latitude":41.45803,"longitude":2.21152,"geohash":"sp3ef74qs0","city":"Santa Coloma de Gramenet","country_code":"ES","country":"Spain"},"sensors":[{"id":113,"ancestry":null,"name":"AMS CCS811 - TVOC","description":"Total Volatile Organic Compounds Digital Indoor Se...","unit":"ppb","created_at":"2022-12-12T11:24:13Z","updated_at":"2022-12-12T11:24:13Z","measurement_id":9,"uuid":"2443ecee-2b04-48a0-bb2e-3a2271ee3c38","value":2960.0,"raw_value":2960.0,"prev_value":null,"prev_raw_value":null},{"id":112,"ancestry":null,"name":"AMS CCS811 - eCO2","description":"Equivalent Carbon Dioxide Digital Indoor Sensor","unit":"ppm","created_at":"2022-12-12T11:24:13Z","updated_at":"2022-12-12T11:24:13Z","measurement_id":8,"uuid":"3008d390-0456-43b2-bb14-bcc2ee334971","value":3055.0,"raw_value":3055.0,"prev_value":null,"prev_raw_value":null},{"id":14,"ancestry":null,"name":"BH1730FVC - Light","description":"Digital Ambient Light Sensor","unit":"lux","created_at":"2022-12-12T11:24:13Z","updated_at":"2022-12-12T11:24:13Z","measurement_id":3,"uuid":"f5f84b48-5f9a-43d2-a63c-6b1c2ed49c84","value":0.0,"raw_value":0.0,"prev_value":null,"prev_raw_value":null},{"id":10,"ancestry":null,"name":"Battery SCK","description":"Custom Circuit","unit":"%","created_at":"2022-12-12T11:24:13Z","updated_at":"2022-12-12T11:24:13Z","measurement_id":10,"uuid":"af1b609a-7eac-4d1c-a1ef-8ac96fd629fa","value":-1.0,"raw_value":-1.0,"prev_value":null,"prev_raw_value":null},{"id":53,"ancestry":null,"name":"ICS43432 - Noise","description":"I2S Digital Mems Microphone with custom Audio Proc...","unit":"dBA","created_at":"2022-12-12T11:24:13Z","updated_at":"2022-12-12T11:24:13Z","measurement_id":4,"uuid":"a1145b1c-4a13-4b7b-8bad-8545bfc56da8","value":29.05,"raw_value":29.05,"prev_value":null,"prev_raw_value":null},{"id":58,"ancestry":null,"name":"MPL3115A2 - Barometric Pressure","description":"Digital Barometric Pressure Sensor","unit":"kPa","created_at":"2022-12-12T11:24:13Z","updated_at":"2022-12-12T11:24:13Z","measurement_id":11,"uuid":"439c22f3-a5a1-4d20-8dc0-99112c2d1831","value":100.81,"raw_value":100.81,"prev_value":null,"prev_raw_value":null},{"id":89,"ancestry":null,"name":"PMS5003 - PM1.0","description":"Particle Matter PM 1","unit":"ug/m3","created_at":"2022-12-12T11:24:13Z","updated_at":"2022-12-12T11:24:13Z","measurement_id":5,"uuid":"033e6fd3-e02d-4d89-91d7-447793a9282d","value":null,"raw_value":null,"prev_value":null,"prev_raw_value":null},{"id":88,"ancestry":null,"name":"PMS5003 - PM10","description":"Particle Matter PM 10","unit":"ug/m3","created_at":"2022-12-12T11:24:13Z","updated_at":"2022-12-12T11:24:13Z","measurement_id":7,"uuid":"cb4266c5-1419-4ccf-a3aa-2da118590e8a","value":null,"raw_value":null,"prev_value":null,"prev_raw_value":null},{"id":87,"ancestry":null,"name":"PMS5003 - PM2.5","description":"Particle Matter PM 2.5","unit":"ug/m3","created_at":"2022-12-12T11:24:13Z","updated_at":"2022-12-12T11:24:13Z","measurement_id":6,"uuid":"d9f3cbc5-722a-47f2-a50c-a77daca06fb0","value":null,"raw_value":null,"prev_value":null,"prev_raw_value":null},{"id":56,"ancestry":null,"name":"SHT31 - Humidity","description":"Humidity","unit":"%","created_at":"2022-12-12T11:24:13Z","updated_at":"2022-12-12T11:24:13Z","measurement_id":2,"uuid":"b57b2c84-4eae-4a0c-9093-ba6f3e249f6a","value":58.88,"raw_value":58.88,"prev_value":null,"prev_raw_value":null},{"id":55,"ancestry":null,"name":"SHT31 - Temperature","description":"Temperature","unit":"ºC","created_at":"2022-12-12T11:24:13Z","updated_at":"2022-12-12T11:24:13Z","measurement_id":1,"uuid":"914bc58a-3313-4539-bc00-1f6affcc1c64","value":27.54,"raw_value":27.54,"prev_value":null,"prev_raw_value":null}]},"kit":{"id":26,"uuid":"c1d08b12-dc87-47d0-94a5-b9ba1cf43dbe","slug":"sck:2,1","name":"SCK 2.1","description":"Smart Citizen Kit 2.1 with Urban Sensor Board","created_at":"2022-12-12T11:26:40Z","updated_at":"2022-12-12T11:29:32Z"}}{"id":1,"uuid":"57fec898-fa09-445c-bc30-5eab0c627a3b","name":"Test","description":null,"state":"has_published","postprocessing":null,"hardware_info":{"id":"8517F6DB5055344E312E3120FF0E1943","mac":"BE:FF:4D:07:25:96","time":"2023-06-11T11:14:59Z","esp_bd":"2021-08-26T16:39:05Z","hw_ver":"2.1","sam_bd":"2022-11-22T16:49:09Z","esp_ver":"0.9.8-e283e58","sam_ver":"0.9.8-739fd4a"},"system_tags":["offline"],"user_tags":[],"is_private":false,"notify_low_battery":false,"notify_stopped_publishing":false,"last_reading_at":"2023-06-12T02:57:08Z","added_at":"2022-12-12T11:56:35Z","updated_at":"2023-06-11T11:15:06Z","mac_address":"[FILTERED]","owner":{"id":1,"uuid":"4505db42-12ba-410c-bc53-89a3ea5b3e92","username":"oscar","avatar":"https://smartcitizen.s3.amazonaws.com/avatars/default.svg","url":null,"joined_at":"2022-12-09T17:58:21Z","location":{"city":null,"country":null,"country_code":null},"device_ids":[1,2]},"data":{"recorded_at":"2023-06-12T02:57:08Z","added_at":"2023-06-12T02:57:08Z","location":{"ip":null,"exposure":null,"elevation":null,"latitude":41.45803,"longitude":2.21152,"geohash":"sp3ef74qs0","city":"Santa Coloma de Gramenet","country_code":"ES","country":"Spain"},"sensors":[{"id":113,"ancestry":null,"name":"AMS CCS811 - TVOC","description":"Total Volatile Organic Compounds Digital Indoor Se...","unit":"ppb","created_at":"2022-12-12T11:24:13Z","updated_at":"2022-12-12T11:24:13Z","measurement_id":9,"uuid":"2443ecee-2b04-48a0-bb2e-3a2271ee3c38","value":2960.0,"raw_value":2960.0,"prev_value":null,"prev_raw_value":null},{"id":112,"ancestry":null,"name":"AMS CCS811 - eCO2","description":"Equivalent Carbon Dioxide Digital Indoor Sensor","unit":"ppm","created_at":"2022-12-12T11:24:13Z","updated_at":"2022-12-12T11:24:13Z","measurement_id":8,"uuid":"3008d390-0456-43b2-bb14-bcc2ee334971","value":3055.0,"raw_value":3055.0,"prev_value":null,"prev_raw_value":null},{"id":14,"ancestry":null,"name":"BH1730FVC - Light","description":"Digital Ambient Light Sensor","unit":"lux","created_at":"2022-12-12T11:24:13Z","updated_at":"2022-12-12T11:24:13Z","measurement_id":3,"uuid":"f5f84b48-5f9a-43d2-a63c-6b1c2ed49c84","value":0.0,"raw_value":0.0,"prev_value":null,"prev_raw_value":null},{"id":10,"ancestry":null,"name":"Battery SCK","description":"Custom Circuit","unit":"%","created_at":"2022-12-12T11:24:13Z","updated_at":"2022-12-12T11:24:13Z","measurement_id":10,"uuid":"af1b609a-7eac-4d1c-a1ef-8ac96fd629fa","value":-1.0,"raw_value":-1.0,"prev_value":null,"prev_raw_value":null},{"id":53,"ancestry":null,"name":"ICS43432 - Noise","description":"I2S Digital Mems Microphone with custom Audio Proc...","unit":"dBA","created_at":"2022-12-12T11:24:13Z","updated_at":"2022-12-12T11:24:13Z","measurement_id":4,"uuid":"a1145b1c-4a13-4b7b-8bad-8545bfc56da8","value":29.05,"raw_value":29.05,"prev_value":null,"prev_raw_value":null},{"id":58,"ancestry":null,"name":"MPL3115A2 - Barometric Pressure","description":"Digital Barometric Pressure Sensor","unit":"kPa","created_at":"2022-12-12T11:24:13Z","updated_at":"2022-12-12T11:24:13Z","measurement_id":11,"uuid":"439c22f3-a5a1-4d20-8dc0-99112c2d1831","value":100.81,"raw_value":100.81,"prev_value":null,"prev_raw_value":null},{"id":89,"ancestry":null,"name":"PMS5003 - PM1.0","description":"Particle Matter PM 1","unit":"ug/m3","created_at":"2022-12-12T11:24:13Z","updated_at":"2022-12-12T11:24:13Z","measurement_id":5,"uuid":"033e6fd3-e02d-4d89-91d7-447793a9282d","value":null,"raw_value":null,"prev_value":null,"prev_raw_value":null},{"id":88,"ancestry":null,"name":"PMS5003 - PM10","description":"Particle Matter PM 10","unit":"ug/m3","created_at":"2022-12-12T11:24:13Z","updated_at":"2022-12-12T11:24:13Z","measurement_id":7,"uuid":"cb4266c5-1419-4ccf-a3aa-2da118590e8a","value":null,"raw_value":null,"prev_value":null,"prev_raw_value":null},{"id":87,"ancestry":null,"name":"PMS5003 - PM2.5","description":"Particle Matter PM 2.5","unit":"ug/m3","created_at":"2022-12-12T11:24:13Z","updated_at":"2022-12-12T11:24:13Z","measurement_id":6,"uuid":"d9f3cbc5-722a-47f2-a50c-a77daca06fb0","value":null,"raw_value":null,"prev_value":null,"prev_raw_value":null},{"id":56,"ancestry":null,"name":"SHT31 - Humidity","description":"Humidity","unit":"%","created_at":"2022-12-12T11:24:13Z","updated_at":"2022-12-12T11:24:13Z","measurement_id":2,"uuid":"b57b2c84-4eae-4a0c-9093-ba6f3e249f6a","value":58.88,"raw_value":58.88,"prev_value":null,"prev_raw_value":null},{"id":55,"ancestry":null,"name":"SHT31 - Temperature","description":"Temperature","unit":"ºC","created_at":"2022-12-12T11:24:13Z","updated_at":"2022-12-12T11:24:13Z","measurement_id":1,"uuid":"914bc58a-3313-4539-bc00-1f6affcc1c64","value":27.54,"raw_value":27.54,"prev_value":null,"prev_raw_value":null}]},"kit":{"id":26,"uuid":"c1d08b12-dc87-47d0-94a5-b9ba1cf43dbe","slug":"sck:2,1","name":"SCK 2.1","description":"Smart Citizen Kit 2.1 with Urban Sensor Board","created_at":"2022-12-12T11:26:40Z","updated_at":"2022-12-12T11:29:32Z"}}Retry later
Retry later
{"id":1,"uuid":"57fec898-fa09-445c-bc30-5eab0c627a3b","name":"Test","description":null,"state":"has_published","postprocessing":null,"hardware_info":{"id":"8517F6DB5055344E312E3120FF0E1943","mac":"BE:FF:4D:07:25:96","time":"2023-06-11T11:14:59Z","esp_bd":"2021-08-26T16:39:05Z","hw_ver":"2.1","sam_bd":"2022-11-22T16:49:09Z","esp_ver":"0.9.8-e283e58","sam_ver":"0.9.8-739fd4a"},"system_tags":["offline"],"user_tags":[],"is_private":false,"notify_low_battery":false,"notify_stopped_publishing":false,"last_reading_at":"2023-06-12T02:57:08Z","added_at":"2022-12-12T11:56:35Z","updated_at":"2023-06-11T11:15:06Z","mac_address":"[FILTERED]","owner":{"id":1,"uuid":"4505db42-12ba-410c-bc53-89a3ea5b3e92","username":"oscar","avatar":"https://smartcitizen.s3.amazonaws.com/avatars/default.svg","url":null,"joined_at":"2022-12-09T17:58:21Z","location":{"city":null,"country":null,"country_code":null},"device_ids":[1,2]},"data":{"recorded_at":"2023-06-12T02:57:08Z","added_at":"2023-06-12T02:57:08Z","location":{"ip":null,"exposure":null,"elevation":null,"latitude":41.45803,"longitude":2.21152,"geohash":"sp3ef74qs0","city":"Santa Coloma de Gramenet","country_code":"ES","country":"Spain"},"sensors":[{"id":113,"ancestry":null,"name":"AMS CCS811 - TVOC","description":"Total Volatile Organic Compounds Digital Indoor Se...","unit":"ppb","created_at":"2022-12-12T11:24:13Z","updated_at":"2022-12-12T11:24:13Z","measurement_id":9,"uuid":"2443ecee-2b04-48a0-bb2e-3a2271ee3c38","value":2960.0,"raw_value":2960.0,"prev_value":null,"prev_raw_value":null},{"id":112,"ancestry":null,"name":"AMS CCS811 - eCO2","description":"Equivalent Carbon Dioxide Digital Indoor Sensor","unit":"ppm","created_at":"2022-12-12T11:24:13Z","updated_at":"2022-12-12T11:24:13Z","measurement_id":8,"uuid":"3008d390-0456-43b2-bb14-bcc2ee334971","value":3055.0,"raw_value":3055.0,"prev_value":null,"prev_raw_value":null},{"id":14,"ancestry":null,"name":"BH1730FVC - Light","description":"Digital Ambient Light Sensor","unit":"lux","created_at":"2022-12-12T11:24:13Z","updated_at":"2022-12-12T11:24:13Z","measurement_id":3,"uuid":"f5f84b48-5f9a-43d2-a63c-6b1c2ed49c84","value":0.0,"raw_value":0.0,"prev_value":null,"prev_raw_value":null},{"id":10,"ancestry":null,"name":"Battery SCK","description":"Custom Circuit","unit":"%","created_at":"2022-12-12T11:24:13Z","updated_at":"2022-12-12T11:24:13Z","measurement_id":10,"uuid":"af1b609a-7eac-4d1c-a1ef-8ac96fd629fa","value":-1.0,"raw_value":-1.0,"prev_value":null,"prev_raw_value":null},{"id":53,"ancestry":null,"name":"ICS43432 - Noise","description":"I2S Digital Mems Microphone with custom Audio Proc...","unit":"dBA","created_at":"2022-12-12T11:24:13Z","updated_at":"2022-12-12T11:24:13Z","measurement_id":4,"uuid":"a1145b1c-4a13-4b7b-8bad-8545bfc56da8","value":29.05,"raw_value":29.05,"prev_value":null,"prev_raw_value":null},{"id":58,"ancestry":null,"name":"MPL3115A2 - Barometric Pressure","description":"Digital Barometric Pressure Sensor","unit":"kPa","created_at":"2022-12-12T11:24:13Z","updated_at":"2022-12-12T11:24:13Z","measurement_id":11,"uuid":"439c22f3-a5a1-4d20-8dc0-99112c2d1831","value":100.81,"raw_value":100.81,"prev_value":null,"prev_raw_value":null},{"id":89,"ancestry":null,"name":"PMS5003 - PM1.0","description":"Particle Matter PM 1","unit":"ug/m3","created_at":"2022-12-12T11:24:13Z","updated_at":"2022-12-12T11:24:13Z","measurement_id":5,"uuid":"033e6fd3-e02d-4d89-91d7-447793a9282d","value":null,"raw_value":null,"prev_value":null,"prev_raw_value":null},{"id":88,"ancestry":null,"name":"PMS5003 - PM10","description":"Particle Matter PM 10","unit":"ug/m3","created_at":"2022-12-12T11:24:13Z","updated_at":"2022-12-12T11:24:13Z","measurement_id":7,"uuid":"cb4266c5-1419-4ccf-a3aa-2da118590e8a","value":null,"raw_value":null,"prev_value":null,"prev_raw_value":null},{"id":87,"ancestry":null,"name":"PMS5003 - PM2.5","description":"Particle Matter PM 2.5","unit":"ug/m3","created_at":"2022-12-12T11:24:13Z","updated_at":"2022-12-12T11:24:13Z","measurement_id":6,"uuid":"d9f3cbc5-722a-47f2-a50c-a77daca06fb0","value":null,"raw_value":null,"prev_value":null,"prev_raw_value":null},{"id":56,"ancestry":null,"name":"SHT31 - Humidity","description":"Humidity","unit":"%","created_at":"2022-12-12T11:24:13Z","updated_at":"2022-12-12T11:24:13Z","measurement_id":2,"uuid":"b57b2c84-4eae-4a0c-9093-ba6f3e249f6a","value":58.88,"raw_value":58.88,"prev_value":null,"prev_raw_value":null},{"id":55,"ancestry":null,"name":"SHT31 - Temperature","description":"Temperature","unit":"ºC","created_at":"2022-12-12T11:24:13Z","updated_at":"2022-12-12T11:24:13Z","measurement_id":1,"uuid":"914bc58a-3313-4539-bc00-1f6affcc1c64","value":27.54,"raw_value":27.54,"prev_value":null,"prev_raw_value":null}]},"kit":{"id":26,"uuid":"c1d08b12-dc87-47d0-94a5-b9ba1cf43dbe","slug":"sck:2,1","name":"SCK 2.1","description":"Smart Citizen Kit 2.1 with Urban Sensor Board","created_at":"2022-12-12T11:26:40Z","updated_at":"2022-12-12T11:29:32Z"}}Retry later

So basically, there is a throttling in between, but it immediately comes back to accepting requests.

[timcowlishaw]

Hey Oscar, I've just been looking at this - I saw the same behaviour, but after a hard rebuild of the docker container it seems to have fixed itself! I'm seeing the throttling working as before for unauthorized users, but. shall we verify together next week before deploying?

docker images seem weirdly 'sticky' btw - i had to do a doco up -d --force-recreate --build app sidekiq mqtt-task telnet-task to get it to pick up the changes after switching branches.

[oscgonfer]

#227