FIWARE datatarget #141

bkd231 · 2022-02-14T22:13:47Z

This PR extends the list of supported datatarget types with integration to the FIWARE NGSI-LD Context Brokers.

DataTargetType enum is extended with FIWARE option
DataTargetKafkaListenerService recognizes datatarget type and invokes send function of a corresponding service
New FiwareDataTarget entity and changes in DataTargetService (CRUD)

The change was developed and tested with the Scorpio Context Broker.

* Deleting application now also deletes LoRaWAN devices in ChirpStack * Removed unused imports in application.service.ts

* Log level is now configurable globally * Read log configuration from Environment Variables * Added comments and fixed indentation. * Minor cleanup in nestjs.ts * Removed unused code and fixed config. Co-authored-by: nlg <nlg@iterator-it.dk>

GufCab

Error when navigating back from editing data target

Navigate to editing a data target
Press the "Back" button
User is presented with 404 error

This also happens on saving a data target after edit.

GufCab · 2022-02-15T11:36:31Z

src/entities/dto/create-data-target.dto.ts

@@ -35,7 +41,7 @@ export class CreateDataTargetDto {
    @IsString()
    @MaxLength(1024)
    @IsNotBlank("url")
-    @IsUrl()
+    @IsUrl({ require_tld: false, require_protocol: true})


Neat - this is a good input

GufCab · 2022-02-15T11:37:12Z

src/entities/fiware-data-target.entity.ts

+
+    @BeforeInsert()
+    private beforeInsert() {
+        /**


Is this a todo/note?

GufCab · 2022-02-15T12:45:15Z

src/services/data-targets/fiware-data-target.service.ts

+            }
+            return this.success(target);
+        } catch (err) {
+            // TODO: Error handling for common errors


Unfixed TODO

GufCab · 2022-02-15T15:01:20Z

src/migration/1644874601012-fiware-data-target.ts

@@ -0,0 +1,26 @@
+import {MigrationInterface, QueryRunner} from "typeorm";


I have some issues with the database changes here.
With the new menu/page for picking data targets, we are opening up for multiple new data targets - which is a great new feature.

This database change just expands the current data_target table with the FIWARE specific fields - this may become an issue later on where we will have to expand the model each time a new data target type is added.

I would rather have a separate table or some other solution that doesn't fill out the table with fields that are only sometimes relevant.

GufCab · 2022-02-15T15:03:23Z

src/services/data-targets/data-target.service.ts

@@ -233,12 +234,18 @@ export class DataTargetService {
            (dataTarget as HttpPushDataTarget).timeout = dataTargetDto.timeout;
            (dataTarget as HttpPushDataTarget).authorizationHeader =
                dataTargetDto.authorizationHeader;
+        } else if (dataTargetDto.type === DataTargetType.Fiware) {


We should probably just do the cast once, but I agree that this is the way that it is currently done - so this is fine.

GufCab · 2022-02-15T15:07:29Z

src/services/data-targets/fiware-data-target.service.ts

+
+     makeAxiosConfiguration(
+        config: FiwareDataTargetConfiguration,
+        data: TransformedPayloadDto


Unused argument

GufCab · 2022-02-16T15:41:10Z

src/services/data-targets/data-target.service.ts

@@ -233,12 +234,18 @@ export class DataTargetService {
            (dataTarget as HttpPushDataTarget).timeout = dataTargetDto.timeout;
            (dataTarget as HttpPushDataTarget).authorizationHeader =
                dataTargetDto.authorizationHeader;
+        } else if (dataTargetDto.type === DataTargetType.Fiware) {


* Feature/IOT_16_MulticastBackend (#132) * Made CRUD operationer for multicast. Tested with frontend. * Made CRUD for multicast in backend plus connection to chirpStack. * Changed chirpstack applicationID since there will always only be one. * Split multicast in two entities so it's easier to expand later. Made a new entity called lorawanMulticastDefinition which will contain the informations about a lorawan multicast * made functionality so devices now will be added to chirpstack if they are a lorawan device. Also made the update functionality, so a device will be removed if it's not a part of the new multicast * Send message. Possible to get current message queue and to overwrite it * Made validation for service profile. Devices should only be added to multicast if they alle have same service profile. * PR changes * PR changes - fixed pagination for multicast * PR Changes * PR Changes * Pr changes Co-authored-by: August Andersen <aha@iterator-it.dk> * Db migrations (#133) * Made migrations. Now it's nessesary to add migrations when changes are made in db. The command - npm run typeorm migration:generate -- -n <migrationName> - will generate a migration file if changes are made compared to the db. When you launch the app, a migration:run command will be called. This will apply the newly migration. If you want to revert a migration, npm run typeorm migration:revert can be called. It will revert the latest migration. If you are in doubt which migrations has been called or not, you can write npm run typeorm migration:show. This will show you the pending/fulfilled migrations. * Since migrations are made in prestart, no need to check on dist. * PR Changes * PR changes Co-authored-by: August Andersen <aha@iterator-it.dk> * Migrations changes in ormconfig file to make migrations possible in test environment * Initial migration (#134) * Initial migration * Fix proper linting ignore of migrations * Changed ormconfig.ts to .js so dist folder is created correctly. Minor changes in package.json. Removed multicast from initialmigration and made a seperate migration with multicast. Co-authored-by: augusthjerrild <augusthjerrild@gmail.com> * Feature/1220 api key (#136) * Init api key auth with hardcoded keys * Added TODOs. Throw 401 if api key is invalid * Fix roles metadata not set on class controller * Fetch api keys and sort. Prepare for create and update * Api key fetch and create done * Cleanup api key flow. Remove update flow for now * Validate api key access * Works - typeerror when building * Fixed circular dependency error * Added API guard to relevant controllers * Fix indentation. Delete unused auth api key request Co-authored-by: Aram Al-Sabti <afa@iterator-it.dk> Co-authored-by: nlg <nlg@iterator-it.dk> * Fix roles in controllers where it was set on the whole class (#139) * Edit API keys (#138) * Add option for editing API key * Fix API keys with admin not having write access * Edit API key PR * Clean up API key * CVE-2019-18413. Patch for potential SQL injections (#137) * CVE-2019-18413. Patch for potential SQL injections * Fix request 400 on get applications by permission * Spell organization with British English ("z") * Simplified migration names * Optimize chirpstack calls when fetching devices (#143) * FIWARE datatarget (#141) * Fiware DataTarget Support * Migration for Fiware Datatarget * Fixing incorrect log message * PR Fixes * Optimize bulk import and the load on chirpstack (#140) * Adjust eslint * Modify bulk import create to take batches. Update missing * Remove restriction on devices belonging to the same application * Optimize chirpstack calls. Init updatemany endpoint. * Implement updateMany and cleanup * Fix device model not set. Cleanup code. Add comments * Refactor iot device helpers * Make device model error code more specific * Added comment every time invalid devices are filtered * Fixed issue when creating new IoT device with no device model * Fixed Fiware datatarget headers declarations and corresponding unit tests (#144) Co-authored-by: August Andersen <aha@iterator-it.dk> Co-authored-by: Aram Al-Sabti <afa@iterator-it.dk> Co-authored-by: nlg <nlg@iterator-it.dk> Co-authored-by: Bartek <88727464+bkdkmd@users.noreply.github.com>

MadsSvejstrup and others added 10 commits December 10, 2021 11:32

Added environment variable for Postgres SSL config (#118)

fcb3eab

Increased timeout for requests against ChirpStack application server (#…

7b9e780

…124)

Issue/47 frontend delete devices when deleting application (#123)

e6faba2

* Deleting application now also deletes LoRaWAN devices in ChirpStack * Removed unused imports in application.service.ts

Pass copies of data to untrusted code instead of references (#125)

16b319b

Issue/120 configurable log levels (#127)

326ced6

* Log level is now configurable globally * Read log configuration from Environment Variables * Added comments and fixed indentation. * Minor cleanup in nestjs.ts * Removed unused code and fixed config. Co-authored-by: nlg <nlg@iterator-it.dk>

Fetch latest message instead of all (#126)

11b6ae8

Fiware DataTarget Support

449e818

Merge branch 'fiware-data-target-sq' into fiware-data-target-stage

ff8d0f5

Migration for Fiware Datatarget

6b7c41b

Fixing incorrect log message

92a38df

bkd231 mentioned this pull request Feb 15, 2022

FIWARE datatarget OS2iot/OS2iot-frontend#67

Merged

GufCab self-requested a review February 15, 2022 14:40

GufCab requested changes Feb 15, 2022

View reviewed changes

PR Fixes

5b59bb8

GufCab approved these changes Feb 17, 2022

View reviewed changes

GufCab merged commit 77862b7 into OS2iot:stage Feb 17, 2022

bkd231 deleted the fiware-data-target-stage branch July 1, 2022 09:51

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

FIWARE datatarget #141

FIWARE datatarget #141

bkd231 commented Feb 14, 2022

GufCab left a comment

GufCab Feb 15, 2022

GufCab Feb 15, 2022

GufCab Feb 15, 2022

bkd231 Feb 16, 2022

GufCab Feb 15, 2022

GufCab Feb 15, 2022

bkd231 Feb 16, 2022

GufCab Feb 16, 2022

GufCab Feb 15, 2022

bkd231 Feb 16, 2022

GufCab Feb 16, 2022

		@@ -0,0 +1,26 @@
		import {MigrationInterface, QueryRunner} from "typeorm";

FIWARE datatarget #141

FIWARE datatarget #141

Conversation

bkd231 commented Feb 14, 2022

GufCab left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment