MEN-4325 File Transfer Limits #21

merlin-northern · 2021-03-12T10:20:11Z

ChangeLog:none
Signed-off-by: Peter Grzybowski peter@northern.tech

mender-test-bot · 2021-03-12T10:20:28Z

@merlin-northern, Let me know if you want to start the integration pipeline by mentioning me and the command "start pipeline".

mender-test-bot · 2021-03-14T20:45:27Z

@merlin-northern, Let me know if you want to start the integration pipeline by mentioning me and the command "start pipeline".

mender-test-bot · 2021-03-14T21:29:54Z

@merlin-northern, Let me know if you want to start the integration pipeline by mentioning me and the command "start pipeline".

mender-test-bot · 2021-03-14T21:35:42Z

@merlin-northern, Let me know if you want to start the integration pipeline by mentioning me and the command "start pipeline".

mender-test-bot · 2021-03-14T21:41:16Z

@merlin-northern, Let me know if you want to start the integration pipeline by mentioning me and the command "start pipeline".

mender-test-bot · 2021-03-14T22:02:48Z

@merlin-northern, Let me know if you want to start the integration pipeline by mentioning me and the command "start pipeline".

mender-test-bot · 2021-03-14T22:30:28Z

@merlin-northern, Let me know if you want to start the integration pipeline by mentioning me and the command "start pipeline".

mender-test-bot · 2021-03-15T08:44:36Z

@merlin-northern, Let me know if you want to start the integration pipeline by mentioning me and the command "start pipeline".

mender-test-bot · 2021-03-15T09:02:13Z

@merlin-northern, Let me know if you want to start the integration pipeline by mentioning me and the command "start pipeline".

mender-test-bot · 2021-03-15T09:27:24Z

@merlin-northern, Let me know if you want to start the integration pipeline by mentioning me and the command "start pipeline".

mender-test-bot · 2021-03-15T09:39:22Z

@merlin-northern, Let me know if you want to start the integration pipeline by mentioning me and the command "start pipeline".

merlin-northern · 2021-03-15T09:45:28Z

@mender-test-bot start pipeline please :)

mender-test-bot · 2021-03-15T09:45:54Z

Hello 😸 I created a pipeline for you here: Pipeline-270502493

Build Configuration Matrix

Key	Value
AUDITLOGS_REV	origin/master
BUILD_BEAGLEBONEBLACK	true
BUILD_CLIENT	true
BUILD_QEMUX86_64_BIOS_GRUB	true
BUILD_QEMUX86_64_BIOS_GRUB_GPT	true
BUILD_QEMUX86_64_UEFI_GRUB	true
BUILD_VEXPRESS_QEMU	true
BUILD_VEXPRESS_QEMU_FLASH	true
BUILD_VEXPRESS_QEMU_UBOOT_UEFI_GRUB	true
CREATE_ARTIFACT_WORKER_REV	origin/master
DEPLOYMENTS_ENTERPRISE_REV	origin/master
DEPLOYMENTS_REV	origin/master
DEVICEAUTH_REV	origin/master
DEVICECONFIG_REV	origin/master
DEVICECONNECT_REV	origin/master
GUI_REV	origin/master
INTEGRATION_REV	master
INVENTORY_ENTERPRISE_REV	origin/master
INVENTORY_REV	origin/master
MENDER_ARTIFACT_REV	origin/master
MENDER_CLI_REV	origin/master
MENDER_CONNECT_REV	pull/21/head
MENDER_REV	origin/master
MTLS_AMBASSADOR_REV	origin/master
RUN_INTEGRATION_TESTS	true
TENANTADM_REV	origin/master
TEST_QEMUX86_64_BIOS_GRUB	true
TEST_QEMUX86_64_BIOS_GRUB_GPT	true
TEST_QEMUX86_64_UEFI_GRUB	true
TEST_VEXPRESS_QEMU	true
TEST_VEXPRESS_QEMU_FLASH	true
TEST_VEXPRESS_QEMU_UBOOT_UEFI_GRUB	true
USERADM_ENTERPRISE_REV	origin/master
USERADM_REV	origin/master
WORKFLOWS_ENTERPRISE_REV	origin/master
WORKFLOWS_REV	origin/master

alfrunes

Mostly suggestions, but the additions to the sessions.UploadHandler must come sooner or the file will be overwritten before reaching those checks.

config/config.go

alfrunes · 2021-03-15T15:25:02Z

config/config.go

@@ -255,6 +307,9 @@ func (c *MenderShellConfig) Validate() (err error) {
 	}

 	c.HTTPSClient.Validate()
+
+	//enforce the limits by default, if set to false it means permit everything
+	c.Limits.Enabled = true


... but won't the default (empty) limitation disable everything?

not really -- thats why it is done like that; e.g.: FollowSymLinks: can be true or false, but the check will run anyways if limits are enabled. I want a central place to switch the limits on and off (similar to selinux enforce/permissive). so, unless you are adamant on this one :)

limits/filetransfer/limits.go

alfrunes · 2021-03-15T15:40:41Z

limits/filetransfer/limits.go

+			sinceLastUpdateS := time.Now().Unix() - deviceCountersLastH.bytesTransferredLastUpdate.Unix()
+			if deviceCountersLastH.bytesTransferred != 0 {
+				deviceCountersLastH.currentTxRate = float64(deviceCountersLastH.bytesTransferred*1.0) / float64(sinceLastUpdateS)
+			}
+			sinceLastUpdateS = time.Now().Unix() - deviceCountersLastH.bytesReceivedLastUpdate.Unix()
+			if deviceCountersLastH.bytesReceived != 0 {
+				deviceCountersLastH.currentRxRate = float64(deviceCountersLastH.bytesReceived*1.0) / float64(sinceLastUpdateS)
+			}


Are these values used for anything except for logging?

those are for "future use" just in case we need to reach for the previous hour. I can remove them, they are leftovers as well.

I changed to hold just an average of last minute, for extra info, not used in the limits atm.

alfrunes · 2021-03-15T16:00:42Z

config/config.go

+	// By default we only allow to send/put regular files
+	RegularFilesOnly bool


I guess this only affects stat requests? Uploads and downloads are already restricted to regular files.

really? I tried with socket and fifo, did not dare to try char/block /dev'ices but I think this limit makes sense to have explicitly.

I was wrong, it does affect downloads. Uploads are however restricted to regular files because of this line:

mender-connect/session/filetransfer.go

Line 471 in 1399abb

err = fd.Chmod(os.FileMode(*params.Mode) & os.ModePerm)

And the way the file is created:

mender-connect/session/filetransfer.go

Line 397 in 1399abb

fd, err = os.OpenFile(filename, os.O_CREATE|os.O_EXCL|os.O_WRONLY, 0200)

config/config.go

alfrunes · 2021-03-15T16:17:54Z

limits/filetransfer/limits.go

+		return nil
+	}
+
+	if (mode & S_ISUID) != 0 {


This will never be the case.

if you send mode=4755 it will

Unless you have the "PreserveModes" config set, only the permission modes counts:

mender-connect/session/filetransfer.go

Line 471 in 1399abb

err = fd.Chmod(os.FileMode(*params.Mode) & os.ModePerm)

limits/filetransfer/limits.go

session/model/filetransfer.go

mender-test-bot · 2021-03-18T21:15:26Z

@merlin-northern, Let me know if you want to start the integration pipeline by mentioning me and the command "start pipeline".

mender-test-bot · 2021-03-18T22:03:20Z

@merlin-northern, Let me know if you want to start the integration pipeline by mentioning me and the command "start pipeline".

mender-test-bot · 2021-03-18T22:21:19Z

@merlin-northern, Let me know if you want to start the integration pipeline by mentioning me and the command "start pipeline".

mender-test-bot · 2021-03-18T22:26:03Z

@merlin-northern, Let me know if you want to start the integration pipeline by mentioning me and the command "start pipeline".

mender-test-bot · 2021-03-18T22:27:15Z

@merlin-northern, Let me know if you want to start the integration pipeline by mentioning me and the command "start pipeline".

mender-test-bot · 2021-03-18T22:32:22Z

@merlin-northern, Let me know if you want to start the integration pipeline by mentioning me and the command "start pipeline".

mender-test-bot · 2021-03-24T21:09:39Z

@merlin-northern, Let me know if you want to start the integration pipeline by mentioning me and the command "start pipeline".

mender-test-bot · 2021-03-25T20:47:28Z

@merlin-northern, Let me know if you want to start the integration pipeline by mentioning me and the command "start pipeline".

alfrunes

I'm pretty sure the transfer-rate computation is incorrect, and I also thinks that section deserves some cleanup. Also, the unit tests job is failing.

alfrunes · 2021-03-29T09:55:45Z

limits/filetransfer/limits.go

+			time.Sleep(time.Duration(countersUpdateSleepTimeS) * time.Second)
+			bytesRXLastMinute += deviceCounters.bytesReceived - bytesRXLastMinute0
+			bytesTXLastMinute += deviceCounters.bytesTransferred - bytesTXLastMinute0
+			deviceCounters.bytesReceivedLast1W = expWeight1m*deviceCounters.bytesReceivedLast1W +


I don't understand what the intention of this counter is "Last 1 Week?".

last 1 minute weighted, sorry. I will change that. I think I changed in one place, this one was left.

alfrunes · 2021-03-29T10:02:51Z

limits/filetransfer/limits.go

+			bytesRXLastMinute += deviceCounters.bytesReceived - bytesRXLastMinute0
+			bytesTXLastMinute += deviceCounters.bytesTransferred - bytesTXLastMinute0


I don't think this is correct.
Every transfer time interval is counted multiple times except the very last.

alfrunes · 2021-03-29T10:24:38Z

limits/filetransfer/limits.go

+func updatePerHourCounters() {
+	const one60th = 0.016666666666666666
+	if counterUpdateRunning {
+		counterUpdateStarted <- false
+		return
+	}
+
+	counterUpdateRunning = true
+	counterUpdateStarted <- true
+	//exponential decrease for "avg" over 1 minute: \exp(-dt[s]/(1*60[s]))
+	expWeight1m := math.Exp(-float64(countersUpdateSleepTimeS) * one60th) // / 60.0)
+	//analogy to the uptime weighted avg
+	//expWeight5m := math.Exp(-float64(countersUpdateSleepTimeS) * 0.003333333333333333)  // / (5*60.0))
+	//expWeight15m := math.Exp(-float64(countersUpdateSleepTimeS) * 0.001111111111111111) // / (15*60.0))
+	deviceCounters.bytesReceivedLast1W = 0.0
+	deviceCounters.bytesTransferredLast1W = 0.0
+
+	for counterUpdateRunning {
+		bytesRXLastMinute0 := deviceCounters.bytesReceived
+		bytesTXLastMinute0 := deviceCounters.bytesTransferred
+		bytesRXLastMinute := uint64(0)
+		bytesTXLastMinute := uint64(0)
+		for i := 0; i < int(int64(time.Minute)/int64(countersUpdateSleepTimeS)/int64(time.Second)); i++ {
+			time.Sleep(time.Duration(countersUpdateSleepTimeS) * time.Second)
+			bytesRXLastMinute += deviceCounters.bytesReceived - bytesRXLastMinute0
+			bytesTXLastMinute += deviceCounters.bytesTransferred - bytesTXLastMinute0
+			deviceCounters.bytesReceivedLast1W = expWeight1m*deviceCounters.bytesReceivedLast1W +
+				float64(bytesRXLastMinute) - expWeight1m*float64(bytesRXLastMinute)
+			deviceCounters.bytesTransferredLast1W = expWeight1m*deviceCounters.bytesTransferredLast1W +
+				float64(bytesTXLastMinute) - expWeight1m*float64(bytesTXLastMinute)
+		}
+		deviceCounters.rateTransferredLastMinute = float64(bytesTXLastMinute) * one60th
+		deviceCounters.rateReceivedLastMinute = float64(bytesRXLastMinute) * one60th
+	}
+}


Can we please simplify this a little? I found it very difficult to read.

Suggested change

func updatePerHourCounters() {

const one60th = 0.016666666666666666

if counterUpdateRunning {

counterUpdateStarted <- false

return

}

counterUpdateRunning = true

counterUpdateStarted <- true

//exponential decrease for "avg" over 1 minute: \exp(-dt[s]/(1*60[s]))

expWeight1m := math.Exp(-float64(countersUpdateSleepTimeS) * one60th) // / 60.0)

//analogy to the uptime weighted avg

//expWeight5m := math.Exp(-float64(countersUpdateSleepTimeS) * 0.003333333333333333) // / (5*60.0))

//expWeight15m := math.Exp(-float64(countersUpdateSleepTimeS) * 0.001111111111111111) // / (15*60.0))

deviceCounters.bytesReceivedLast1W = 0.0

deviceCounters.bytesTransferredLast1W = 0.0

for counterUpdateRunning {

bytesRXLastMinute0 := deviceCounters.bytesReceived

bytesTXLastMinute0 := deviceCounters.bytesTransferred

bytesRXLastMinute := uint64(0)

bytesTXLastMinute := uint64(0)

for i := 0; i < int(int64(time.Minute)/int64(countersUpdateSleepTimeS)/int64(time.Second)); i++ {

time.Sleep(time.Duration(countersUpdateSleepTimeS) * time.Second)

bytesRXLastMinute += deviceCounters.bytesReceived - bytesRXLastMinute0

bytesTXLastMinute += deviceCounters.bytesTransferred - bytesTXLastMinute0

deviceCounters.bytesReceivedLast1W = expWeight1m*deviceCounters.bytesReceivedLast1W +

float64(bytesRXLastMinute) - expWeight1m*float64(bytesRXLastMinute)

deviceCounters.bytesTransferredLast1W = expWeight1m*deviceCounters.bytesTransferredLast1W +

float64(bytesTXLastMinute) - expWeight1m*float64(bytesTXLastMinute)

}

deviceCounters.rateTransferredLastMinute = float64(bytesTXLastMinute) * one60th

deviceCounters.rateReceivedLastMinute = float64(bytesRXLastMinute) * one60th

}

}

func updateTransferRates() {

if counterUpdateRunning {

counterUpdateStarted <- false

return

}

counterUpdateRunning = true

counterUpdateStarted <- true

var (

lastRX uint64

lastTX uint64

avgRX float64

avgTX float64

// exp factor: 1 - exp(-dt[s]/(1*60[s]))

_w1 = math.Exp(-(float64(countersUpdateSleepTimeS) * (1.0 / 60.0)))

w1 = 1 - _w1

)

tick := time.NewTicker(time.Duration(countersUpdateSleepTimeS) * time.Second)

defer tick.Stop()

for counterUpdateRunning {

select {

case <-tick.C:

rxTot := deviceCounters.bytesTransferred

txTot := deviceCounters.bytesReceived

rx := rxTot - lastRX

tx := txTot - lastTX

lastRX = rxTot

lastTX = rxTot

// avg[n+1] = w * Y[n] + (1 - w) avg[n]

avgRX = w1*float64(rx) + _w1*avgRX

avgTX = w1*float64(tx) + _w1*avgTX

deviceCounters.rateTransferredLastMinute = avgTX

deviceCounters.rateReceivedLastMinute = avgRX

// case <-ctx.Done(): ?

}

}

}

sure thing.

alfrunes · 2021-03-29T10:36:09Z

config/config.go

+	// By default we only allow to send/put regular files
+	RegularFilesOnly bool


I was wrong, it does affect downloads. Uploads are however restricted to regular files because of this line:

mender-connect/session/filetransfer.go

Line 471 in 1399abb

err = fd.Chmod(os.FileMode(*params.Mode) & os.ModePerm)

And the way the file is created:

mender-connect/session/filetransfer.go

Line 397 in 1399abb

fd, err = os.OpenFile(filename, os.O_CREATE|os.O_EXCL|os.O_WRONLY, 0200)

mender-test-bot · 2021-03-29T15:36:07Z

@merlin-northern, Let me know if you want to start the integration pipeline by mentioning me and the command "start pipeline".

merlin-northern · 2021-03-29T15:43:50Z

@mender-test-bot start pipeline and I beg you let it pass. sugar pretty please with cherry on top.

mender-test-bot · 2021-03-29T15:44:16Z

Hello 😸 I created a pipeline for you here: Pipeline-278324343

Build Configuration Matrix

Key	Value
AUDITLOGS_REV	origin/master
BUILD_BEAGLEBONEBLACK	true
BUILD_CLIENT	true
BUILD_QEMUX86_64_BIOS_GRUB	true
BUILD_QEMUX86_64_BIOS_GRUB_GPT	true
BUILD_QEMUX86_64_UEFI_GRUB	true
BUILD_VEXPRESS_QEMU	true
BUILD_VEXPRESS_QEMU_FLASH	true
BUILD_VEXPRESS_QEMU_UBOOT_UEFI_GRUB	true
CREATE_ARTIFACT_WORKER_REV	origin/master
DEPLOYMENTS_ENTERPRISE_REV	origin/master
DEPLOYMENTS_REV	origin/master
DEVICEAUTH_REV	origin/master
DEVICECONFIG_REV	origin/master
DEVICECONNECT_REV	origin/master
GUI_REV	origin/master
INTEGRATION_REV	master
INVENTORY_ENTERPRISE_REV	origin/master
INVENTORY_REV	origin/master
MENDER_ARTIFACT_REV	origin/master
MENDER_CLI_REV	origin/master
MENDER_CONNECT_REV	pull/21/head
MENDER_REV	origin/master
MTLS_AMBASSADOR_REV	origin/master
RUN_INTEGRATION_TESTS	true
TENANTADM_REV	origin/master
TEST_QEMUX86_64_BIOS_GRUB	true
TEST_QEMUX86_64_BIOS_GRUB_GPT	true
TEST_QEMUX86_64_UEFI_GRUB	true
TEST_VEXPRESS_QEMU	true
TEST_VEXPRESS_QEMU_FLASH	true
TEST_VEXPRESS_QEMU_UBOOT_UEFI_GRUB	true
USERADM_ENTERPRISE_REV	origin/master
USERADM_REV	origin/master
WORKFLOWS_ENTERPRISE_REV	origin/master
WORKFLOWS_REV	origin/master

mender-test-bot · 2021-03-29T21:15:56Z

@merlin-northern, Let me know if you want to start the integration pipeline by mentioning me and the command "start pipeline".

merlin-northern · 2021-03-29T21:16:45Z

@mender-test-bot start pipeline pretty please

mender-test-bot · 2021-03-29T21:17:11Z

Hello 😸 I created a pipeline for you here: Pipeline-278468367

Build Configuration Matrix

Key	Value
AUDITLOGS_REV	origin/master
BUILD_BEAGLEBONEBLACK	true
BUILD_CLIENT	true
BUILD_QEMUX86_64_BIOS_GRUB	true
BUILD_QEMUX86_64_BIOS_GRUB_GPT	true
BUILD_QEMUX86_64_UEFI_GRUB	true
BUILD_VEXPRESS_QEMU	true
BUILD_VEXPRESS_QEMU_FLASH	true
BUILD_VEXPRESS_QEMU_UBOOT_UEFI_GRUB	true
CREATE_ARTIFACT_WORKER_REV	origin/master
DEPLOYMENTS_ENTERPRISE_REV	origin/master
DEPLOYMENTS_REV	origin/master
DEVICEAUTH_REV	origin/master
DEVICECONFIG_REV	origin/master
DEVICECONNECT_REV	origin/master
GUI_REV	origin/master
INTEGRATION_REV	master
INVENTORY_ENTERPRISE_REV	origin/master
INVENTORY_REV	origin/master
MENDER_ARTIFACT_REV	origin/master
MENDER_CLI_REV	origin/master
MENDER_CONNECT_REV	pull/21/head
MENDER_REV	origin/master
MTLS_AMBASSADOR_REV	origin/master
RUN_INTEGRATION_TESTS	true
TENANTADM_REV	origin/master
TEST_QEMUX86_64_BIOS_GRUB	true
TEST_QEMUX86_64_BIOS_GRUB_GPT	true
TEST_QEMUX86_64_UEFI_GRUB	true
TEST_VEXPRESS_QEMU	true
TEST_VEXPRESS_QEMU_FLASH	true
TEST_VEXPRESS_QEMU_UBOOT_UEFI_GRUB	true
USERADM_ENTERPRISE_REV	origin/master
USERADM_REV	origin/master
WORKFLOWS_ENTERPRISE_REV	origin/master
WORKFLOWS_REV	origin/master

alfrunes · 2021-03-30T06:55:51Z

limits/filetransfer/limits.go

+	}
+}
+
+func updatePerHourCounters() {


The function name is a bit imprecise.

crap! and I so renamed all the structs. fixing. but look, I renamed the test func :)

ChangeLog:none Signed-off-by: Peter Grzybowski <peter@northern.tech>

mender-test-bot · 2021-03-30T07:04:42Z

@merlin-northern, Let me know if you want to start the integration pipeline by mentioning me and the command "start pipeline".

merlin-northern force-pushed the men_4325_file_transfer_restrictions_review branch from e4ffc62 to 44eaaad Compare March 14, 2021 20:45

merlin-northern force-pushed the men_4325_file_transfer_restrictions_review branch from 44eaaad to ec4969f Compare March 14, 2021 21:29

merlin-northern force-pushed the men_4325_file_transfer_restrictions_review branch from ec4969f to f0b6310 Compare March 14, 2021 21:35

merlin-northern force-pushed the men_4325_file_transfer_restrictions_review branch from f0b6310 to 0fc5ada Compare March 14, 2021 21:40

merlin-northern force-pushed the men_4325_file_transfer_restrictions_review branch from 0fc5ada to 58fe03a Compare March 14, 2021 22:02

merlin-northern force-pushed the men_4325_file_transfer_restrictions_review branch from 58fe03a to db2fa9f Compare March 14, 2021 22:30

merlin-northern force-pushed the men_4325_file_transfer_restrictions_review branch from db2fa9f to a39ef22 Compare March 15, 2021 08:44

merlin-northern force-pushed the men_4325_file_transfer_restrictions_review branch from a39ef22 to 2569263 Compare March 15, 2021 09:01

merlin-northern force-pushed the men_4325_file_transfer_restrictions_review branch from 2569263 to b84f5a0 Compare March 15, 2021 09:27

merlin-northern force-pushed the men_4325_file_transfer_restrictions_review branch from b84f5a0 to d9b6307 Compare March 15, 2021 09:39

merlin-northern requested a review from alfrunes March 15, 2021 09:45

alfrunes requested changes Mar 15, 2021

View reviewed changes

merlin-northern force-pushed the men_4325_file_transfer_restrictions_review branch from 6ab9208 to 642aea8 Compare March 18, 2021 22:32

merlin-northern force-pushed the men_4325_file_transfer_restrictions_review branch from cf60df5 to 80a0ba7 Compare March 25, 2021 20:47

merlin-northern requested a review from alfrunes March 25, 2021 20:47

alfrunes requested changes Mar 29, 2021

View reviewed changes

merlin-northern force-pushed the men_4325_file_transfer_restrictions_review branch from 80a0ba7 to 7a28c6f Compare March 29, 2021 15:35

merlin-northern requested a review from alfrunes March 29, 2021 15:43

merlin-northern force-pushed the men_4325_file_transfer_restrictions_review branch from 7a28c6f to f0ef192 Compare March 29, 2021 21:15

alfrunes approved these changes Mar 30, 2021

View reviewed changes

MEN-4325 File Transfer Limits

1b72a8a

ChangeLog:none Signed-off-by: Peter Grzybowski <peter@northern.tech>

merlin-northern force-pushed the men_4325_file_transfer_restrictions_review branch from f0ef192 to 1b72a8a Compare March 30, 2021 07:04

merlin-northern merged commit 7f567d2 into mendersoftware:master Mar 30, 2021

		// By default we only allow to send/put regular files
		RegularFilesOnly bool

		bytesRXLastMinute += deviceCounters.bytesReceived - bytesRXLastMinute0
		bytesTXLastMinute += deviceCounters.bytesTransferred - bytesTXLastMinute0

MEN-4325 File Transfer Limits #21

MEN-4325 File Transfer Limits #21

Conversation

merlin-northern commented Mar 12, 2021

mender-test-bot commented Mar 12, 2021

mender-test-bot commented Mar 14, 2021

mender-test-bot commented Mar 14, 2021

mender-test-bot commented Mar 14, 2021

mender-test-bot commented Mar 14, 2021

mender-test-bot commented Mar 14, 2021

mender-test-bot commented Mar 14, 2021

mender-test-bot commented Mar 15, 2021

mender-test-bot commented Mar 15, 2021

mender-test-bot commented Mar 15, 2021

mender-test-bot commented Mar 15, 2021

merlin-northern commented Mar 15, 2021

mender-test-bot commented Mar 15, 2021

alfrunes left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

merlin-northern Mar 17, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mender-test-bot commented Mar 18, 2021

mender-test-bot commented Mar 18, 2021

mender-test-bot commented Mar 18, 2021

mender-test-bot commented Mar 18, 2021

mender-test-bot commented Mar 18, 2021

mender-test-bot commented Mar 18, 2021

mender-test-bot commented Mar 24, 2021

mender-test-bot commented Mar 25, 2021

alfrunes left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mender-test-bot commented Mar 29, 2021

merlin-northern commented Mar 29, 2021

mender-test-bot commented Mar 29, 2021

mender-test-bot commented Mar 29, 2021

merlin-northern commented Mar 29, 2021

mender-test-bot commented Mar 29, 2021

Choose a reason for hiding this comment

merlin-northern Mar 30, 2021 • edited Loading

Choose a reason for hiding this comment

mender-test-bot commented Mar 30, 2021

merlin-northern Mar 17, 2021 •

edited

Loading

merlin-northern Mar 30, 2021 •

edited

Loading