Support hierarchical categories

[exp]

Ki-nTree is designed with the idea of a top level parent category, and then a subcategory. This is a little restrictive and doesn't match how my categories are set up in Inventree.

I've prepared a patch to implement hierarchical categories for the setup function and the user category select screen, but before proposing a PR I thought I'd open an Issue for discussion.

I don't understand why Digikey's categories are merged with the user's category list in this screen, nor the rationale behind integration of categories with Kicad. I think there are a few valid options for how these could be handled, so if you are open to adding this feature could you let me know how you'd like it integrated best?

Thanks.

[eeintech]

Hi there @exp

Thanks for your feedback, and yes you are completely right, the full tree implementation is missing... your patch would be a welcome addition!

The reason I implemented the automatic matching between suppliers categories and InvenTree categories was to make the process of adding a part to InvenTree faster than manual one, either by recalling a previous relationship (stored in yaml files) or by partial word matching between the two category names.
For KiCad, it is only required to automatically pre-fill the symbol/template/footprint libraries form, step required to add the part to KiCad's libaries. This is again to shave time in the process of adding a new part.

While you may consider those as "nice-to-have" features, I personally do enjoy their convenience. I haven't had anybody not wanting them so I would prefer keeping those. If your PR keeps those fully functional I wouldn't mind merging.

If you need more in-depth explanation about the automatic match feature, let me know.

Thanks!

[eeintech]

Hello @exp

Did you have more questions regarding the use of categories in Ki-nTree?

[exp]

Hi @eeintech no that's fine, I am just trying to find time to get this coded. I'm having troubles receiving from Digikey and I would like to add a search results box too. I'll break the commits up appropriately.

I should have time to work on it this week, but I have to finish the actual PCBs first ☺

[eeintech]

Ok sounds good, good luck with the PCBs!

[eeintech]

Hi @exp How are the PCBs doing? Are you still planning to support this issue with a PR?

[exp]

Hi @eeintech I will get to this issue as soon as possible. I've had issues with getting any results at all from Digikey which makes it quite difficult to test and I've had to jump to other projects. Sorry for the delay.

[eeintech]

No problem, thanks for the update!

[eeintech]

@exp Any progress? 😄

[martonmiklos]

Hi folks,

Once upon a time I was thinking about making a 'supplier API' -> Inventree conversion program, but luckily I found Kin-tree before starting the actual coding.

My original idea was to provide an Inventree category tree selector to the user in the part add dialog.

When a part is added to a specific category in Inventree from a supplier category then the match would be saved to a database (SQLite for e.g.). If a category match found in this db for a new part the Inventree category would have been already selected in the new part dialog.
Over time this would have greatly reduced the category selection task.

[eeintech]

@martonmiklos I have implemented some kind of automatic matching process in the category selection, either by:

• matching to supplier categories
• remembering the user choice on the last supplier category match

Check out the get_categories method:

Ki-nTree/kintree/database/inventree_interface.py

Line 45 in 253f248

def get_categories(part_info: dict, supplier_only=False) -> list:

[eeintech]

@exp What implementation scheme did you have in mind? How would you improve the categories window?

[Weissnix4711]

Just my two cents, but how about copying the category dialog from Inventree?

Instead of two seperate dropdowns for category and subcategory, have one dropdown which lists all available categories, subcategories, sub-subcategories, etc. Only problem is the custom category name option. You could again copy Inventree, and have a dedicated 'Add category' form, or just let the user type their own full category path. Eg, Capacitors/Aluminium/SMD. You'd need to parse that somehow, figure out which (sub)categories already exist, and create the subcategories as necessary.

[exp]

Hi All,
I apologise for my tardiness on this issue. My PCB project is still a high priority and I took some time off inbetween. This week I am free to work on these projects. I want to get a patch into Inventree but then I am fine with any approach.

@eeintech I planned to make the application a single unified window where possible, with a drop-down as Weissnix suggests. My inventree has quite extensive categorisation so I want to make sure they're fully supported as deep as possible, it's the matching of supplier categories I need to take a bit of time to do.

Sorry again for the delay,
Cheers!

[eeintech]

Just my two cents, but how about copying the category dialog from Inventree?

Instead of two seperate dropdowns for category and subcategory, have one dropdown which lists all available categories, subcategories, sub-subcategories, etc. Only problem is the custom category name option. You could again copy Inventree, and have a dedicated 'Add category' form, or just let the user type their own full category path. Eg, Capacitors/Aluminium/SMD. You'd need to parse that somehow, figure out which (sub)categories already exist, and create the subcategories as necessary.

Ha, I like this idea! So simple and efficient, no reinventing the wheel.

@exp Let me know where can I can give you a hand, I would like to get this implemented soon 😄

[exp]

@eeintech I ideally need to address the Inventree issue I opened regarding part boxes and storage first but I appreciate I've delayed you plenty on this. If you want to take the lead I'd really appreciate that but if not I'll try and contribute some examples in the next day.

[eeintech]

@exp I picked up some workload recently and have to prioritize this, as soon as I am done I wish to look into this, if you haven't already 😃

[Fivefold]

It's probably obvious to @eeintech but I just want to note that for proper hierarchical category support the structure of both categories.yaml and supplier_parameters.yaml would need some changes.

1. Currently (if my interpretation is correct, please correct me if I'm wrong) the "codes" (the abbreviations for the inventree-internal part number, e.g. TRA for transistors) only support top-level categories.
   

   Ki-nTree/kintree/config/inventree/categories.yaml

   Lines 2 to 18 in 84be9dd

   	CODES:
   	Assemblies: PCA
   	Capacitors: CAP
   	Circuit Protections: PRO
   	Connectors: CON
   	Crystals and Oscillators: CLK
   	Diodes: DIO
   	Inductors: IND
   	Integrated Circuits: ICS
   	Mechanicals: MEC
   	Miscellaneous: MIS
   	Modules: MOD
   	Power Management: PWR
   	Printed-Circuit Boards: PCB
   	RF: RFC
   	Resistors: RES
   	Transistors: TRA

   
   A solution would be to match the keys in the code section with categories, regardless of nesting level. Of course this would prevent two different categories with the same name to have different codes, but that seems like an unlikely use-case.

2. Categories would also need an overhaul
   

   Ki-nTree/kintree/config/inventree/categories.yaml

   Lines 20 to 23 in 84be9dd

   	CATEGORIES:
   	Assemblies:
   	- Printed-Circuit Board Assembly
   	- Product

   
   Here the easiest change I could think of would be something like this:

CATEGORIES:
  Passive Components: # Level 0
    Capacitors:       # Level 1
      Ceramic:        # Level 2
      Electrolytic:   # Level 2
    Inductors:        # Level 1
    Resistors:        # Level 1

Of course this mixes values and keys depending on context so there are probably better solutions. I'm no yaml guru unfortunately.

3. In supplier_parameters.yaml there doesn't seem to be support for subcategories even now. Direct name matching like in 1. would probably be the easiest way (for the user).

[eeintech]

@Fivefold There is indeed a little bit of thinking for deeper category trees.

1. Currently (if my interpretation is correct, please correct me if I'm wrong) the "codes" (the abbreviations for the inventree-internal part number, e.g. TRA for transistors) only support top-level categories.

That's correct and if user picks a sub-category, the code used is from its parent.

2. Categories would also need an overhaul

Right, it could be updated to a dictionary based on InvenTree categories.

3. In supplier_parameters.yaml there doesn't seem to be support for subcategories

Indeed to lower the complexity, parameters are shared across top-level categories, and not sub-categories.

Here are some thoughts, dreaming a tiny bit bigger:

• Category tree fetched from InvenTree directly, with an expiration date to avoid fetching every single search
• Auto-update to categories.yaml file, move codes to a separate one
• In category selection window: propose existing InvenTree categories in InvenTree format (see @Weissnix4711 proposal above)
• Give user option to enter a new category (eg. ICs/Logic/And Gate) and create it on the fly, if user rights permit
• Auto-fetch and auto-update of supplier_parameters.yaml based on InvenTree categories parameters, with an expiration date to avoid fetching every single search
• Extra GUI step/window showing a table containing: supplier parameter name, supplier parameter value, dropdown with exisiting InvenTree categoy parameters and extra field for creating a new category parameter. If a match was already found, use it as default value inside dropdown box. If extra field is filled in: create parameter in InvenTree and ignore dropdown value.

[martonmiklos]

I started to work on this, I have a dynamically populated (from InvenTree) treeview:

I plan to add a recent list next to it with the latest imported categories.

[eeintech]

@martonmiklos Nice work! Any chance you could share your implementation with us?

[martonmiklos]

@eeintech The code lives here:
https://github.com/martonmiklos/Ki-nTree/tree/rething_category_selection

It is above a TME support branch, but I plan to rebase to master when I am done.

[eeintech]

@martonmiklos Thanks, I'll try to take a look soon 😃

[martonmiklos]

@martonmiklos Thanks, I'll try to take a look soon smiley

Okay just for sync up here is my vision on how should it look like (in the case if InvenTree export is selected):

Ki-n-tree has an InvenTree PK - supplier category identifier lookup
User enters the partnumber, data queried from the supplier
If the supplier category has match in the lookup then in the second page under here:

We should write the category path from InvenTree plus an Edit button.
In this case if the user hit the submit button the part shall be created without any further dialogs.

In the case if no mapping found this line shall be hidden and when the user hits the submit the category selection dialog shall be shown next.

If there is an existing mapping in the recent list this shall be the topmost item, and shall be selected by default.
The same dialog shall be opened in the case if the user selects the Edit button next to the Category path.

[eeintech]

We should write the category path from InvenTree plus an Edit button.

That's a good idea actually, I haven't thought of merging the part form window with category window.

In the case if no mapping found this line shall be hidden and when the user hits the submit the category selection dialog shall be shown next.

If I'm not mistaken, that would be two ways to select categories with two different flows (first is nested inside the part form window and second is after the part form window, as it is now). I'm not sure why you want those two flows... I would prefer to stick to your original idea and, when there is no match, using the edit button would be the only way to select the category.
The validation of the form, instead of being currently done only on "Name" and "Description", would also be done on the category selection. I think it would simplify the flow so there is only one entry point for the category selection.
What do you think?

The Edit category window you showed looks cool, is it functional?
If it is, I have some time on my hands right now, I could finish the implementation if you'd like.
We could merge you branch into 0.6.x and it would add your changes to the current opened PR.

[eeintech]

@martonmiklos I'm thinking to first merge and cut the 0.6.0 release soon as it brings a bunch of new stuff. Then work on the category revamp. What do you think?

[martonmiklos]

The improved category selection is heavily WIP, and I have no idea when will I find some time to finish. So my recommendation is to go ahead, I will handle the rebase on my fork.

[martonmiklos]

I apologize I forgot to reply to this one.

If I'm not mistaken, that would be two ways to select categories with two different flows (first is nested inside the part form window and second is after the part form window, as it is now). I'm not sure why you want those two flows... I would prefer to stick to your original idea and, when there is no match, using the edit button would be the only way to select the category. The validation of the form, instead of being currently done only on "Name" and "Description", would also be done on the category selection. I think it would simplify the flow so there is only one entry point for the category selection. What do you think?

You might misunderstood me. My plan is to make a single window to select the category, and it would allow two way for category selection: tree based approach from InvenTree and selection from the recent list.

The current flow is :
Enter supplier PN -> Edit supplier data -> select category -> create part

In the case if there is category match it is reduced to:
Enter supplier PN -> Edit supplier data -> create part

If there is no category match then the current flow is being executed.

The Edit category window you showed looks cool, is it functional? If it is, I have some time on my hands right now, I could finish the implementation if you'd like. We could merge you branch into 0.6.x and it would add your changes to the current opened PR.

The tree selection side is functional the recent list is not yet. From my side this week is quite busy, if you can move things forward on my work feel free to do so. I can catch up with it next week.

[eeintech]

Hello @martonmiklos @exp

I have gotten feedback on the category selector and have found multiple issue with old code which I want to fix now and release in a newer version of Ki-nTree (thinking 0.7).
That's why I would like to go ahead on this implementation you started @martonmiklos in this branch?
https://github.com/martonmiklos/Ki-nTree/tree/rething_category_selection

[martonmiklos]

Hey,

Uhm yes. I messed up that branch a bit as I used to...
It contains 3 commits from the master.

• Remember last supplier -> this had been merged
• Support for TME import. I wanted to PR it, but I wanted to have some discussion on this before
• Aand the the category selector changes in the question

Before moving forward I need to separate these things, preferably rebase the TME and merge that first.

[eeintech]

@martonmiklos What is TME? Happy to discuss it 👍

If you do some clean-up let me know!

[eeintech]

FYI I have implemented it in the new GUI, target is 1.0.0 release:

[hvraven]

Hi @eeintech ,
thanks for implementing this. I am currently testing the new 1.0.0b1 release and it looks really nice. I am however not able to configure the parameters_filters.yaml and supplier categories (digikey_categories.yaml) correctly to make them work with hierachical categories. We store other components in Inventree and thus have all electronics below a top level Electronics category. I tried nesting the categories in the yaml, as well as using forward slashes in the name directly.
Is this supposed to be working already? If you could give me a pointer on where to start looking I could also try to help with the implementation.

[eeintech]

Hello @hvraven

Actually I was thinking to deprecate the Supplier category lookups for 1.0.0 as it is quite easier to select a category now...

The parameters_filters.yaml file works for me, it will only take effect with Digi-Key searches at the moment as other suppliers parameter mapping is missing. Do you have the problem with Digi-Key? Can you please show me the output of Ki-nTree in your terminal?

Also do you mind sharing a overall glimpse at your category mapping (you don't have to share the full file, just let me know the structure so I can try to reproduce)?

Thanks!