Optimize bulk import and the load on chirpstack #140
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
AramAlsabti
changed the title
GufCab
requested changes
Feb 15, 2022
GufCab
approved these changes
Feb 17, 2022
GufCab
approved these changes
Feb 17, 2022
GufCab
added a commit
that referenced
this pull request
Feb 22, 2022
* Feature/IOT_16_MulticastBackend (#132) * Made CRUD operationer for multicast. Tested with frontend. * Made CRUD for multicast in backend plus connection to chirpStack. * Changed chirpstack applicationID since there will always only be one. * Split multicast in two entities so it's easier to expand later. Made a new entity called lorawanMulticastDefinition which will contain the informations about a lorawan multicast * made functionality so devices now will be added to chirpstack if they are a lorawan device. Also made the update functionality, so a device will be removed if it's not a part of the new multicast * Send message. Possible to get current message queue and to overwrite it * Made validation for service profile. Devices should only be added to multicast if they alle have same service profile. * PR changes * PR changes - fixed pagination for multicast * PR Changes * PR Changes * Pr changes Co-authored-by: August Andersen <aha@iterator-it.dk> * Db migrations (#133) * Made migrations. Now it's nessesary to add migrations when changes are made in db. The command - npm run typeorm migration:generate -- -n <migrationName> - will generate a migration file if changes are made compared to the db. When you launch the app, a migration:run command will be called. This will apply the newly migration. If you want to revert a migration, npm run typeorm migration:revert can be called. It will revert the latest migration. If you are in doubt which migrations has been called or not, you can write npm run typeorm migration:show. This will show you the pending/fulfilled migrations. * Since migrations are made in prestart, no need to check on dist. * PR Changes * PR changes Co-authored-by: August Andersen <aha@iterator-it.dk> * Migrations changes in ormconfig file to make migrations possible in test environment * Initial migration (#134) * Initial migration * Fix proper linting ignore of migrations * Changed ormconfig.ts to .js so dist folder is created correctly. Minor changes in package.json. Removed multicast from initialmigration and made a seperate migration with multicast. Co-authored-by: augusthjerrild <augusthjerrild@gmail.com> * Feature/1220 api key (#136) * Init api key auth with hardcoded keys * Added TODOs. Throw 401 if api key is invalid * Fix roles metadata not set on class controller * Fetch api keys and sort. Prepare for create and update * Api key fetch and create done * Cleanup api key flow. Remove update flow for now * Validate api key access * Works - typeerror when building * Fixed circular dependency error * Added API guard to relevant controllers * Fix indentation. Delete unused auth api key request Co-authored-by: Aram Al-Sabti <afa@iterator-it.dk> Co-authored-by: nlg <nlg@iterator-it.dk> * Fix roles in controllers where it was set on the whole class (#139) * Edit API keys (#138) * Add option for editing API key * Fix API keys with admin not having write access * Edit API key PR * Clean up API key * CVE-2019-18413. Patch for potential SQL injections (#137) * CVE-2019-18413. Patch for potential SQL injections * Fix request 400 on get applications by permission * Spell organization with British English ("z") * Simplified migration names * Optimize chirpstack calls when fetching devices (#143) * FIWARE datatarget (#141) * Fiware DataTarget Support * Migration for Fiware Datatarget * Fixing incorrect log message * PR Fixes * Optimize bulk import and the load on chirpstack (#140) * Adjust eslint * Modify bulk import create to take batches. Update missing * Remove restriction on devices belonging to the same application * Optimize chirpstack calls. Init updatemany endpoint. * Implement updateMany and cleanup * Fix device model not set. Cleanup code. Add comments * Refactor iot device helpers * Make device model error code more specific * Added comment every time invalid devices are filtered * Fixed issue when creating new IoT device with no device model * Fixed Fiware datatarget headers declarations and corresponding unit tests (#144) Co-authored-by: August Andersen <aha@iterator-it.dk> Co-authored-by: Aram Al-Sabti <afa@iterator-it.dk> Co-authored-by: nlg <nlg@iterator-it.dk> Co-authored-by: Bartek <88727464+bkdkmd@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When importing a .csv file with a large number of IoT devices, timeouts are very likely to occur. The main suspect is the chirpstack API which cannot keep up as numerous calls are made to the API just for a single LoRaWAN device.
This PR refactors this flow:
Ideally, the backend would process the CSV file. However, that would mean the user doesn't get "live" updates
Related frontend PR
Fixes #130