examples: Externalize configuration values and allow for multiple env… #28

Herrie82 · 2019-11-29T12:12:59Z

…ironments

In order to more easily manage testing of various order types as well as production, development etc. environments, externalize this config values.

Signed-off-by: Herman van Hazendonk github.com@herrie.org

…ironments In order to more easily manage testing of various order types as well as production, development etc. environments, externalize this config values. Signed-off-by: Herman van Hazendonk <github.com@herrie.org>

Herrie82 · 2019-11-29T12:14:18Z

@nanov Happy to receive feedback on how to do things differently. I'm not a very experienced NodeJS developer, so I can see this work and did it based on some "best practice" examples I could find. But open for improvements. Just I'm running Test and Production in parallel and found that it was a bit cumbersome to run the examples the way they were structured currently, hence this PR.

You would basically run the scripts like "node bankLetter.js testing" or "node bankLetter.js production" and it would pick up the correct values for the testing or production server and fallback to testing in case there's an invalid value.

nanov · 2019-11-29T12:35:55Z

Hey @Herrie82, again thank you for your devoted work!

In this case I would structure things a little bit differently and certainly won't add another dependency ( lodash in this case ). I hope i can find the time during the weekend to structure your ideas, which are absolutely in the right direction, into some reusable code.

Herrie82 · 2019-11-29T12:57:29Z

@nanov Yeah I wasn't too happy with the dependency as well. Happy to see what you come up with. As usual there are multiple ways to solve a problem. Mine is one, I'm sure you probably have an even more elegant one ;)

Will be using it shortly in a production system, so trying to contribute some of the things I use and things I ran into to be not very efficient while testing. All little bits always help is my experience, to make it easier for others down the line.

nanov · 2019-11-29T13:08:57Z

Yeah I wasn't too happy with the dependency as well. Happy to see what you come up with. As usual there are multiple ways to solve a problem. Mine is one, I'm sure you probably have an even more elegant one ;)

Well, there are also some personal style preferences and experience involved. At the end of the day working solution is what one is after and yours is one!

Will be using it shortly in a production system, so trying to contribute some of the things I use and things I ran into to be not very efficient while testing. All little bits always help is my experience, to make it easier for others down the line.

We are using this library in production with almost zero issues for a while now.
In order to bring this into a production level fail-safe manner one would have to take some things into consideration on the actual application.

I would do my best to do some more real-world examples over the weekend ( or next week ), as I truly believe it would be useful!

Herrie82 · 2019-12-03T10:34:51Z

@nanov Had any chance to look into this? We're about to deploy into production shortly, so it would be good if we can use your version directly ;)

nanov · 2019-12-04T09:55:29Z

Hi, sorry for the late response, I've been crazy busy these days. I''l do my best to commit something ASAP.

Herrie82 · 2019-12-04T11:29:26Z

@nanov I know the feeling, don't worry. Looking forward to see your solution!

Herrie82 · 2019-12-18T09:00:05Z

@nanov I saw you merged it, but didn't change anything by the looks of it (yet)? Do you plan to make changes still? Just curious. I might be using this now for multiple banks going forward, since it seems to be working well. So I might propose some more changes to be able to deal with that as well in terms of config.

nanov · 2019-12-18T09:02:11Z

@Herrie82 finally found the time to make the changes yes, making them right now, will commit and release later on today!

Herrie82 · 2019-12-18T09:03:40Z

@nanov Ah great. I assume that when using multiple banks we'd need to have separate files to save the keys for each bank? Didn't give much thought about it yet, but since you're making changes you might have some thoughts on that as well and do it in 1 go.

nanov · 2019-12-18T09:08:10Z

Well, how to implement your full architecture is up to you, I'm trying to structure some basic example to get people going.

I could give you some insights on how we operate with mulitple banks, we have a DB record for each ebics acocunt ( ie bank ), and a directory structure so one would know where to look for traces keys and etc.

Also this record contains some settings regarding the account ( STA or C53, special adapters for the data if needed and so on ).

We execute operations based on the account ( for example /accounts/my-cool-ebics-account/downloadDailyStatement/ ), then based on the account config ( from db ) we do the desired operation and process it's results.

Herrie82 · 2019-12-18T11:54:27Z

@nanov OK doesn't sound too distant from what I have for processing individual bank account. Just don't have the whole connectivity bits integrated in our SQLite DB (yet). We have separate scripts for separate channels (FTP, EBICS etc). Good to get your thoughts on that.

nanov · 2019-12-18T14:56:01Z

could you take a look and test the changes?

Herrie82 · 2019-12-18T15:43:35Z

@nanov Ooh nice cleanup again :) I'll give it a go probably tomorrow or Friday. A bit hectic these days.

Herrie82 · 2019-12-24T13:00:45Z

@nanov Sorry for the delay, finally got to test this now. I seem to get the following error. I'm not very familiar with the way you coded it, so just posting it here for guidance:

`C:\node_modules\ebics-client2\examples>node send-vmk-order.js production
Loading config form C:\node_modules\ebics-client2\examples\config with env set to production.
C:\node_modules\ebics-client2\examples\getClient.js:14
} = loadConfig()) => new Client({
^

TypeError: loadConfig is not a function
at module.exports (C:\node_modules\ebics-client2\examples\getClient.js:14:5)
at Object. (C:\node_modules\ebics-client2\examples\send-vmk-order.js:5:38)
at Module._compile (internal/modules/cjs/loader.js:959:30)
at Object.Module._extensions..js (internal/modules/cjs/loader.js:995:10)
at Module.load (internal/modules/cjs/loader.js:815:32)
at Function.Module._load (internal/modules/cjs/loader.js:727:14)
at Function.Module.runMain (internal/modules/cjs/loader.js:1047:10)
at internal/main/run_main_module.js:17:11`

nanov · 2019-12-30T07:37:30Z

my bad, should be fixed right now

Herrie82 · 2019-12-30T08:29:06Z

@nanov better but still not working:

C:\node_modules\ebics-client2\examples>node send-vmk-order.js production
Loading config form C:\node_modules\ebics-client2\examples\config with env set to production.
C:\node_modules\ebics-client2\lib\Client.js:67
throw new Error('EBICS URL is required');
^

Error: EBICS URL is required
at new Client (C:\node_modules\ebics-client2\lib\Client.js:67:10)
at module.exports (C:\node_modules\ebics-client2\examples\getClient.js:14:22)
at Object. (C:\node_modules\ebics-client2\examples\send-vmk-order.js:5:38)
at Module._compile (internal/modules/cjs/loader.js:959:30)
at Object.Module._extensions..js (internal/modules/cjs/loader.js:995:10)
at Module.load (internal/modules/cjs/loader.js:815:32)
at Function.Module._load (internal/modules/cjs/loader.js:727:14)
at Function.Module.runMain (internal/modules/cjs/loader.js:1047:10)
at internal/main/run_main_module.js:17:11

nanov · 2019-12-30T08:50:09Z

just change the 'serverAddress' in the config file to 'url'

nanov · 2019-12-30T08:53:20Z

To be clear, this is just an example how one may build multi config cli with the library

Herrie82 · 2019-12-30T08:55:45Z

@nanov Nope that doesn't work :S

Loading config form C:\node_modules\ebics-client2\examples\config with env set to production.
C:\node_modules\ebics-client2\lib\Client.js:67
throw new Error('EBICS URL is required');
^

Error: EBICS URL is required
at new Client (C:\node_modules\ebics-client2\lib\Client.js:67:10)
at module.exports (C:\node_modules\ebics-client2\examples\getClient.js:14:22)
at Object. (C:\node_modules\ebics-client2\examples\send-vmk-order.js:5:38)
at Module._compile (internal/modules/cjs/loader.js:959:30)
at Object.Module._extensions..js (internal/modules/cjs/loader.js:995:10)
at Module.load (internal/modules/cjs/loader.js:815:32)
at Function.Module._load (internal/modules/cjs/loader.js:727:14)
at Function.Module.runMain (internal/modules/cjs/loader.js:1047:10)
at internal/main/run_main_module.js:17:11

Config file:

{ "url": "https://ebics.gazprombank.lu/ebicsweb/ebicsweb", "partnerId": "XXXXX", "userId": "YYYYY", "hostId": "GPBIEBIX", "passphrase": "MyPassword", "keyStorage": "./keys-prod", "bankName":"Bank GPB International S.A.", "languageCode":"en" }

nanov · 2019-12-30T09:07:42Z

now? :)

Herrie82 · 2019-12-30T09:17:51Z

@nanov Getting closer and closer:

client.send(ebics.Orders.VMK(null, null)) // startDate 'YYYY-MM-DD', endDate 'YYYY-MM-DD'
^

ReferenceError: ebics is not defined

When I manually add:
"const ebics = require('../index');" in the send-vmk-order.js I get the following error:

Loading config form C:\node_modules\ebics-client2\examples\config with env set to production. Error: No keys provided. Can not send the order or any other order for that matter. at Client.send (C:\node_modules\ebics-client2\lib\Client.js:97:28) at processTicksAndRejections (internal/process/task_queues.js:93:5)

nanov · 2019-12-30T09:26:48Z

have you provided keys?

** EDIT **

keyStorage path should be absolute path, or relative to cwd ( ie where you run node )

EDIT 2

Now i saw and fixed the name of the property ( keyStorage -> keyStoragePath )

Herrie82 · 2019-12-30T09:29:11Z

@nanov Yeah I have the "keys-prod"-file in the config directory and in the examples one.

nanov · 2019-12-30T09:29:51Z

See my edits

Herrie82 · 2019-12-30T09:32:40Z

@nanov Yes just leaves the "client.send(ebics.Orders.VMK(null, null)) // startDate 'YYYY-MM-DD', endDate 'YYYY-MM-DD'
^

ReferenceError: ebics is not defined"

nanov · 2019-12-30T09:33:33Z

i'll add those to all examples now, thank you!

nanov · 2019-12-30T09:36:29Z

Could you check it out now?

Herrie82 · 2019-12-30T09:41:55Z

@nanov Yes that seems to work now, thanks :)

nanov · 2019-12-30T14:29:25Z

@Herrie82, could you share your case in more detail, i mean as needs and uses so we( together with you) could that building an ecosystem upon it

Herrie82 · 2019-12-30T14:56:44Z

@nanov for now it's pretty straight forward for EBICS for now. 1 bank, just collecting STA and VMK on scheduled intervals. We run this on a Windows server with simple task scheduler. Works fine in general.

The files get saved to a network share (I added this value for network location to the config file) and simply added writing of the STA/VMK orders to the example files.

From there we feed it (together with files which we got from various other banks via good old S(FTP)) into some custom scripts (VBScript, very old (2008-2012 era), but work very reliably) which has a SQLite DB in the back-end with all our accounts, entity info, daily statements expected etc, so it can verify that for all accounts we have received statements, how many etc. Any special actions that need to be performed on the statements (i.e. account number replacements or some regexp for processing) all happen there. As output we have a concatenated file per account with all statements for that account.

Which we then consolidate into a single file per entity which we send to our accounting and TMS system.

The VBScript also produces some email report about the processing so that we know right away the status in the morning. In case of new accounts these would be added to the DB automatically as well based on found files and would be listed in the email as well, so we know there are new accounts which need "setup" in the database.

Herrie82 · 2020-01-09T10:57:36Z

@nanov We recently got our 2nd bank connection (Credit Suisse), which was on the tested bank list already. Went smoothly. Just wondering how to structure the config files to allow for multiple banks. Any thoughts?

A question occurred in terms of the bank-keys file. Can you use a single file for multiple banks or you would need to have a single keys file per bank?

nanov · 2020-01-14T15:24:32Z

@Herrie82 sorry for the late response!

Can you use a single file for multiple banks or you would need to have a single keys file per bank?
No you can't you'll have to have separate file per bank.

Our setup is something like this:

/ebics-data/{account-name-id}/
   traces/ <- here we save the treaces
   keys/key.key <- the keys
   statements/
      daily/

and so on, as for config you'll have to find a solution that works for you ( loads the right config according to parameter for example ), one option is to do a config format as following : config.{env}.{account}.json and tweak the code a bit, but this is just and option.

nanov closed this Nov 29, 2019

nanov reopened this Nov 29, 2019

nanov merged commit fcbf4ca into node-ebics:master Dec 18, 2019

Herrie82 mentioned this pull request Mar 25, 2021

Examples: Improve config implementation for multibank #51

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

examples: Externalize configuration values and allow for multiple env… #28

examples: Externalize configuration values and allow for multiple env… #28

Herrie82 commented Nov 29, 2019

Herrie82 commented Nov 29, 2019 •

edited

nanov commented Nov 29, 2019

Herrie82 commented Nov 29, 2019 •

edited

nanov commented Nov 29, 2019

Herrie82 commented Dec 3, 2019

nanov commented Dec 4, 2019

Herrie82 commented Dec 4, 2019

Herrie82 commented Dec 18, 2019

nanov commented Dec 18, 2019

Herrie82 commented Dec 18, 2019

nanov commented Dec 18, 2019

Herrie82 commented Dec 18, 2019

nanov commented Dec 18, 2019

Herrie82 commented Dec 18, 2019

Herrie82 commented Dec 24, 2019

nanov commented Dec 30, 2019

Herrie82 commented Dec 30, 2019

nanov commented Dec 30, 2019

nanov commented Dec 30, 2019

Herrie82 commented Dec 30, 2019 •

edited

nanov commented Dec 30, 2019

Herrie82 commented Dec 30, 2019

nanov commented Dec 30, 2019 •

edited

Herrie82 commented Dec 30, 2019

nanov commented Dec 30, 2019

Herrie82 commented Dec 30, 2019

nanov commented Dec 30, 2019

nanov commented Dec 30, 2019

Herrie82 commented Dec 30, 2019

nanov commented Dec 30, 2019

Herrie82 commented Dec 30, 2019

Herrie82 commented Jan 9, 2020

nanov commented Jan 14, 2020

examples: Externalize configuration values and allow for multiple env… #28

examples: Externalize configuration values and allow for multiple env… #28

Conversation

Herrie82 commented Nov 29, 2019

Herrie82 commented Nov 29, 2019 • edited

nanov commented Nov 29, 2019

Herrie82 commented Nov 29, 2019 • edited

nanov commented Nov 29, 2019

Herrie82 commented Dec 3, 2019

nanov commented Dec 4, 2019

Herrie82 commented Dec 4, 2019

Herrie82 commented Dec 18, 2019

nanov commented Dec 18, 2019

Herrie82 commented Dec 18, 2019

nanov commented Dec 18, 2019

Herrie82 commented Dec 18, 2019

nanov commented Dec 18, 2019

Herrie82 commented Dec 18, 2019

Herrie82 commented Dec 24, 2019

nanov commented Dec 30, 2019

Herrie82 commented Dec 30, 2019

nanov commented Dec 30, 2019

nanov commented Dec 30, 2019

Herrie82 commented Dec 30, 2019 • edited

nanov commented Dec 30, 2019

Herrie82 commented Dec 30, 2019

nanov commented Dec 30, 2019 • edited

Herrie82 commented Dec 30, 2019

nanov commented Dec 30, 2019

Herrie82 commented Dec 30, 2019

nanov commented Dec 30, 2019

nanov commented Dec 30, 2019

Herrie82 commented Dec 30, 2019

nanov commented Dec 30, 2019

Herrie82 commented Dec 30, 2019

Herrie82 commented Jan 9, 2020

nanov commented Jan 14, 2020

Herrie82 commented Nov 29, 2019 •

edited

Herrie82 commented Nov 29, 2019 •

edited

Herrie82 commented Dec 30, 2019 •

edited

nanov commented Dec 30, 2019 •

edited