[RFC] API plug-ins

[set-soft]

We all know that KiCost is saturating KitSpace gateway to Octopart. In order to reduce its load I would like to introduce alternative mechanisms. The most important distributors provides some kind of API to do stock queries. I personally use Digi-Key so the first API I'm trying is this.

I created a branch called api_ext_plugin (https://github.com/hildogjr/KiCost/tree/api_ext_plugin). In this branch I'm starting to play with the concept.

The first plug-in is here: Digi-Key plug-in

The setup is a little bit complex, but doing it you get 1000 queries a day for free (slow queries by the way).

Any comments? Anybody can help to test it?

[hildogjr]

Thanks by personally hit me, I didn't see this thread.

Yes, I also had a plan to do that when I re-written some code at distributors folder in the way that any api_*.py / scrap_*py would be automatically imported on __init__.py. I also personally interested on Digikey API (and maybe Mouser).

I did some work passing all general/template to the class at distributor.py and the distributor definitions (since, in this approach, some APIs could return more than one distributor) to distributors_info.py.

We should:

1. Be concerned/create some form of store the user tokens for each of those APIs, may use some '.kicost` file on project folder or OS "application memory" (The GUI does something in this sense, but I think it is away to decrepet);

2. Prioritize the own API answer, e.g.: in case of Digikey and Octopart APIs actived on KiCost, all Digikey spreadsheet information must use it own API as source.

Since this new approach may need some change in files or new input arguments do KiCost, we may start a branch and I hope release a version when we merge to master...

[set-soft]

Hi @hildogjr

The api_ext_plugin branch has some working code, please take a look.

About:

Be concerned/create some form of store the user tokens for each of those APIs, may use some '.kicost` file on project folder or OS "application memory" (The GUI does something in this sense, but I think it is away to decrepet)

I think this should be delegated to the plug-in. Currently kicost-digikey-api-v3 stores the tokens in a configurable path, the default is ~/.config/kicost_digikey_api_v3/.

On KiCost side I wrote a small plug-in wrapper that tries to import the plug-in module. If available gets enabled.

Why try to keep it separated? Many reasons:

• License. The Digi-Key API code is GPL.
• Dependencies. It pulls various dependencies.

I agree we must create a mechanism to detect which methode to use when more than one plug-in can solve the same distributor. But I think this can be achieved using the priorities (already implemented) and one more detail: once a plug-in successfully solved a distributor it should be removed from the list.

[hildogjr]

Agree with your observations.

I made a quick look at the code. Apparently fails on from kicost_digikey_api_v3 import foo is this related with https://pypi.org/project/digikey-api/? (I think is missing some file)

I would rename api_partinfo_digikey_pi.py to simple api_digikey.py.

[hildogjr]

There is the kicost --setup command that I created to integrate KiCost into KiCad and other configurations.

We could use it to the user define which APIs to use and them tokens.

[set-soft]

I made a quick look at the code. Apparently fails on from kicost_digikey_api_v3 import foo is this related with https://pypi.org/project/digikey-api/? (I think is missing some file)

It works here:

$ pytest-3 --log-cli-level debug -k test_acquire_PWM_dk
==================================================================================================================================================== test session starts =====================================================================================================================================================
platform linux -- Python 3.9.2, pytest-6.0.2, py-1.10.0, pluggy-0.13.0
rootdir: /.../KiCost
plugins: xdist-2.2.0, forked-1.3.0
collected 63 items / 62 deselected / 1 selected                                                                                                                                                                                                                                                                              

tests/test_kicost.py::test_acquire_PWM_dk 
------------------------------------------------------------------------------------------------------------------------------------------------------- live log call --------------------------------------------------------------------------------------------------------------------------------------------------------
DEBUG    root:test_kicost.py:340 Test name: acquire_PWM_dk
DEBUG    root:test_kicost.py:69 Testing using: src/kicost --debug 10 --disable_api KitSpace --enable_api Digi-Key -o /.../KiCost/tests/acquire_PWM_dk.xlsx -wi /.../KiCost/tests/acquire-PWM
DEBUG    root:test_kicost.py:246 Converting to CSV
DEBUG    root:test_kicost.py:69 Comparing using: diff -u /.../KiCost/tests/expected_test/acquire_PWM_dk.csv /.../KiCost/tests/result_test/acquire_PWM_dk.csv
DEBUG    root:test_kicost.py:74 Converting to TXT
DEBUG    root:test_kicost.py:69 Comparing using: diff -u /.../KiCost/tests/expected_test/acquire_PWM_dk.xlsx.txt /.../KiCost/tests/result_test/acquire_PWM_dk.xlsx.txt
INFO     root:test_kicost.py:336 acquire_PWM_dk OK
PASSED                                                                                                                                                                                                                                                                                                                 [100%]

============================================================================================================================================== 1 passed, 62 deselected in 2.19s ==============================================================================================================================================

But I think you are confusing the plug-in, I'm not using https://pypi.org/project/digikey-api/ but my own adaptation, look the link in the first post.

I installed the plug-in using pip install -e . from its repo, I can see the link:

$ ls ~/.local/lib/python3.9/site-packages/kicost-digikey-api-v3.egg-link 
/home/salvador/.local/lib/python3.9/site-packages/kicost-digikey-api-v3.egg-link

And the module can be imported without problems:

$ python3
Python 3.9.2 (default, Feb 28 2021, 17:03:44) 
[GCC 10.2.1 20210110] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import kicost_digikey_api_v3
>>> from kicost_digikey_api_v3 import by_digikey_pn, by_manf_pn, by_keyword, configure  # noqa: E402
>>> 

And we have one test using it that is passing. Even for Python 2.7.

[set-soft]

Hi @hildogjr !

I would rename api_partinfo_digikey_pi.py to simple api_digikey.py.

Ok, I renamed it this way.

[set-soft]

There is the kicost --setup command that I created to integrate KiCost into KiCad and other configurations.

We could use it to the user define which APIs to use and them tokens.

Not sure. If this is a plug-in the setup should be done during the plug-in installation.

The current procedure is a little bit complex.

[set-soft]

Hi @hildogjr !

2. Prioritize the own API answer, e.g.: in case of Digikey and Octopart APIs actived on KiCost, all Digikey spreadsheet information must use it own API as source.

I implemented it. Now the Digi-Key API doesn't need to be manually enabled. If the plug-in is present the code will try to use it to solve "Digi-Key". If the plug-in is installed, but the initialization fails the code will print a warning. If the API solves the "Digi-Key" stuff then no other API will try to ask for "Digi-Key" information.

In the test I'm enabling the use of the plug-in (providing a valid configuration) and then asking for Digi-Key only (--include digikey). The values are solved from the cache and then KitSpace is skipped because we solved all the distributors.

[set-soft]

IMPORTANT!!

While implementing this plug-in I realized that we should implement a cache for all the APIs. The cache for Digi-Key defaults to 24 hs TTL, you don't usually need to refresh prices more than once a day.

Implementing it for KitSpace will solve saturation problems without needing to disable the prices by default.

What do you think @hildogjr @mdeweerd ?

[hildogjr]

I still have to try with Digikey account.

Cache will be nice, mainly because we return to the BOM generation process when release that some part is missing. This need appeared some time ago in other thread but, at the time, I didn't think I good way to implement it (Also because, in the model that KiCost is taking, with multiple APIs, this cache should be a top level caching query of all the APIs/scrape/...).

We also should think about the configurations/parameters of KiCost. I think the command line is taking to much options, making difficult to new users.

[set-soft]

Cache will be nice, mainly because we return to the BOM generation process when release that some part is missing. This need appeared some time ago in other thread but, at the time, I didn't think I good way to implement it (Also because, in the model that KiCost is taking, with multiple APIs, this cache should be a top level caching query of all the APIs/scrape/...).

I'm not sure about implementing the cache at top-level. I think each API should implement it. I agree we could have some common code to avoid repetition.

[set-soft]

We also should think about the configurations/parameters of KiCost. I think the command line is taking to much options, making difficult to new users.

Yes, and I don't think KiCost should centralize the complexity of configuring each plug-in. The Digi-Key plug-in will have some configuration details that aren't currently part of KiCost. I.e. country where you place the order, or destination country. And these values doesn't have to be the same for each distributor.

I think the plug-in should allow configuring it in a file. May be we could add some GUI for the file, but this is very plug-in specific.

[mdeweerd]

I am in favor of caching, this may also depend on the API's caching rules/agreement. (For geolocations, Google does not allow caches older than 30 days).

I think caching can be longer than 1 day by default (maybe 1 week). Availability changes faster than prices (especially nowadays), so close to ordering one may want to get the most current stock states. I'ld suggest the default to be about one week and to allow for an option like '--cache-age=0.25' to seek new values when they are older than 6 hours or non-existing.
The cache also needs to keep some history about references that were not found to avoid looking for them on each execution.
Local cache could persist forever regardless of this option, and the spreadsheet may need to show the validity of the values somehow (a cell indicating the oldest date for the data?).

Personnally I have these steps:

1. Ensure that the BOM is valid:
   • valid manufacturers/component codes vs. descriptions;
   • valid variant setups;
2. Get an idea of pricing.
3. Check availability when ordering (unfortunately the alternative part numbers in the BOM are not checked).

This makes me think of another feature: currently kicost does not allow me to check the descriptions from the distributors vs. my own. For instance, I expect 10k, and due to a copy/paste the manufacturer code is for 22k - kicost does not help me detect this unless I click on the distributor links.

[hildogjr]

@set-soft , should it be viable a single .kicost file on default path or in the execution path to keep all the information (for example using YAML language with description and section for each API / sub module).

@mdeweerd, a also did this in a package error once. I would like to use #4 in this sense, by using some "electric grammar interpreter" (https://kitspace.github.io/electro-grammar/) to valid value and footprint, marking the spreadsheet is not match.

[mdeweerd]

Electrogrammar is interesting, but having the possibility to see the description of one of the distributors as text next to the design file description is already helpful (not always, but when option is activated).

I tried the following on Electrogrammar: "r0603 10uF 0.1% 100V" . I do not know if it indicates the inconsistency in its API. I also tried "0603 10uF 0.1% 100V low esr". It works pretty well and could improve the detection of differences.

[set-soft]

@set-soft , should it be viable a single .kicost file on default path or in the execution path to keep all the information (for example using YAML language with description and section for each API / sub module).

This can be done. Currently Linux apps stores the configurations in ~/.config/APP/FILES. What about ~/.config/kicost/api.yaml?

If we want to centralize it we should also make the plug-ins able to use ~/.config/kicost/API_NAME/ directories, for cache and important data (i.e. tokens).

For other OSs we must determine which directory is suitable. I saw a lot of Windows applications using ~/Documents/APP/FILES, it looks strange to me, but a lot of them are using it.

BTW: wxWidgets uses some particular mechanism, and the GUI is currently using it, and with a very archaic format (INI files).

[mdeweerd]

Regarding configurations in ~/.config/kicost/ - I am fine with such an approach for customisations of colors, etc of an UI, and other personal data (username, token, login, etc). However, when it comes to configurations that determine the outcome for a HW project, I prefer keeping it in the project directory.

Using ~/.config/kicost is ok for me for keeping templates that are used for initial project setup.

[set-soft]

This makes me think of another feature: currently kicost does not allow me to check the descriptions from the distributors vs. my own. For instance, I expect 10k, and due to a copy/paste the manufacturer code is for 22k - kicost does not help me detect this unless I click on the distributor links.

Related to this @mdeweerd : In KiBot I'm generating an extra worksheet containing the "Specs" information, is customizable, here is an example:

[hildogjr]

This could be add as additional spreadsheet-page or other KiCost output file... I just have to define how.

About the configuration file, I tend to choose some unique kicost.yml file in the configuration default folder that could have its configuration overwritten by a .kicost.yml on the local project folder (path of KiCost call/execution).

About the cache, K-inTree or other Python package could be used to make it more transparent to KiCost code?

[set-soft]

About the cache, K-inTree or other Python package could be used to make it more transparent to KiCost code?

Take a look at the cache mechanism I'm using, is trivial, no need to add extra dependencies.

[set-soft]

This makes me think of another feature: currently kicost does not allow me to check the descriptions from the distributors vs. my own. For instance, I expect 10k, and due to a copy/paste the manufacturer code is for 22k - kicost does not help me detect this unless I click on the distributor links.

A very small patch: abced19 allows this:

I know this isn't the best, but is a very small change.

[set-soft]

A better implementation: 505ae8d

You get:

Which is what the code wanted to do, but lacked the data

[set-soft]

And now 3880660

Is currently disabled and only enabled by modifying the code, but will be available when an option is added.

[mdeweerd]

Nice!
What about an optional column "Descriptions" next to "Description" combining those of the distributors with &""&... . Goal: fast visual check to match our own description with all the distributor descriptions and detect that either is wrong because of a forgotten "digikey#" incompatible with the "manf#" for instance.

[set-soft]

Nice!
What about an optional column "Descriptions" next to "Description" combining those of the distributors with &""&... . Goal: fast visual check to match our own description with all the distributor descriptions and detect that either is wrong because of a forgotten "digikey#" incompatible with the "manf#" for instance.

Not sure about it because:

1. This is currently supported only by Digi-Key plug-in
2. Mixing global and distributor data doesn't look good

Don't get me wrong, I'm not saying this is a bad idea, just that I won't implement it soon. Perhaps @hildogjr wants to implement it.

[mdeweerd]

2. Mixing global and distributor data doesn't look good

The idea is to generate it only for verification, not for final bom creation.

Don't get me wrong, I'm not saying this is a bad idea, just that I won't implement it soon. Perhaps @hildogjr wants to implement it.

Ok, I guess I can also hide the other columns so that the two interesting ones are still next to each other ;-).

[set-soft]

@hildogjr : replying to myself ;-)

@set-soft , should it be viable a single .kicost file on default path or in the execution path to keep all the information (for example using YAML language with description and section for each API / sub module).

This can be done. Currently Linux apps stores the configurations in ~/.config/APP/FILES. What about ~/.config/kicost/api.yaml?

If we want to centralize it we should also make the plug-ins able to use ~/.config/kicost/API_NAME/ directories, for cache and important data (i.e. tokens).

For other OSs we must determine which directory is suitable. I saw a lot of Windows applications using ~/Documents/APP/FILES, it looks strange to me, but a lot of them are using it.

BTW: wxWidgets uses some particular mechanism, and the GUI is currently using it, and with a very archaic format (INI files).

I implemented it in ~/.config/kicost/config.yaml examples of valid configuration files can be found here: tests/configs.

The cache files are stored in ~/.cache/kicost/

Currently only APIs take advantage of the configuration file. And to be honest just Digi-Key and Octopart really uses it.

Please take a look at it and give me your opinion.

Next will be to add some cache support for KitSpace and Octopart.

BTW: Now KitSpace isn't disabled when Octopart is enabled. This is because Octopart has more priority and we shouldn't annoy KitSpace because we should get all solved using Octopart and nothing will remain.

[set-soft]

More news:

• now you can ask for international prices. I added an example to ask for prices in Spain.
• KitSpace and Octopart cache implemented!! (needs testing)
• Mouser API is also supported

[set-soft]

After a long time:

• Farnell/Newark/CPC API available

@hildogjr : do you think is time to merge it and debug?

[hildogjr]

Sorry be the delay. I am attempting a new job position, I will be quite busy next months.

The branch have passed the automatic tests. If you can confirm, I think will be fine to merge and release (mark the Gittag "vX.X").

[set-soft]

Sorry be the delay. I am attempting a new job position, I will be quite busy next months.

The branch have passed the automatic tests. If you can confirm, I think will be fine to merge and release (mark the Gittag "vX.X").

Ok, I added the TME API and will revisit the Digi-Key API before the merge.

[set-soft]

I merged, but didn't release yet.

I would like to see reports from people using it. The code passes all tests for all Python versions.

The Digi-Key plug-in now is available from PyPi and is a regular dependency. I also tested that KiCost runs without the plugin installed.

But again, I would like to see success reports before releasing it.

[set-soft]

These APIs are in use since 1.1.8 and looks like they are in use because we got 3 reports for problems using Mouser API for people using languages other than english.
I'm closing this.

[mik1234mc]

Confirming that Digikey, Mouser, Farnell and TME APIs are working in our case. Thanks for your effort!