Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Discussion] Adding parts and manipulating stock w/ API #677

Closed
maholli opened this issue Mar 25, 2020 · 26 comments
Closed

[Discussion] Adding parts and manipulating stock w/ API #677

maholli opened this issue Mar 25, 2020 · 26 comments
Labels
barcode Barcode scanning and integration enhancement This is an suggested enhancement or new feature

Comments

@maholli
Copy link

maholli commented Mar 25, 2020

I wanted to get this discussion started and hopefully avoid myself or anyone else having to reinvent the wheel here.

The usefulness of an inventory management tool like InvenTree is directly correlated to its ease of use. Too tedious and folks wont use it.

Hobbyist and Academic Makerspace Use-cases

I use InvenTree for my home inventory and am starting to integrate it into an academic makerspace. My most common activities with the tool are as follows:

  1. Adding new digikey/mouser/lcsc part
  2. Consuming (soldering) a random part or two from inventory and adjusting stock levels accordingly.

Problem

These two activities (especially 1) are extremely tedious. As far as I can tell, the expected workflow is outlined below.

For adding a part:

  1. Pickup loose digikey/mouser/lcsc parts bag.
  2. Read description from bag
  3. Ctrl-F InvenTree Parts tab and look for part
    • Part exists? -> go to Step 4 below.
    • Part doesn't exist? -> create new part
      • Data for new part is entered by hand
  4. After a Part exists, go to Stock tab
    • Select New Stock Item
      • Enter info by hand, select "New Supplier Part"
        • For new supplier part, select newly made part again
        • Open new browser tab -> google distributor name + SKU
        • Transcribe distributor info into New Supplier Part fields
    • Enter appropriate stock quantity and location
  5. Finish
    [There are little time savers here-and-there (like saving a couple button clicks by entering this info into a CSV and importing it).]

I'm aware of the TOS limitations of the various distributors (as discussed in #541 and #566), but there has got to be a better way. Take the digikey API for example, if the individual registers for a free API account, it appears they could use QR/barcode and Product retrieval API while adhering to the TOS.

Envisioned Improvements

What I would like to do is use my phone with the InvenTree app to scan bag barcodes. These activities would take place in a "Parts Cache" tab in InvenTree. The "Parts Cache" would be concurrent between phone/desktop and therefore would allow the user to batch-scan items that can be curated on a desktop, or just scan a single item and act upon it.

Part Actions

Once scanned, the barcode data would be used to retrieve part information from the relevant distributor. The part info is then used to generate a line-item in the "Parts Cache." Next to this line-item would be quick actions that are dependent on whether or not this particular barcode has been entered into InvenTree or not:

  • Barcode has been entered before
    • quick actions could include things like stock +1 / -1 buttons, move stock, new stock, remove stock.
  • Barcode has not been entered before
    • quick actions could include things like new part + supplier part, new supplier part, new part only
    • would be nice for the line-item to also be expandable letting you manually edit common fields before submitting.
    • would also be great to setup "REGEX" rules that allow specific part info strings to be automatically replaced/edited before submitting.

Path Forward

I have a fairly streamlined digikey API workflow in python for scanning barcodes, retrieving part/invoice/pricing info, and populating a spreadsheet (android).

  • I could use help with the barcode scanning into the SQlite db
    ⭐ Since phone browsers can now easily access hardware like cameras...

    Is the best approach is to implement barcode scanning from the browser?

    This would allow laptops, desktops with webcams, and phones to all act as input sources.

Interested in hearing everyone's thoughts.

@SchrodingersGat
Copy link
Member

Hi @maholli thanks for starting this discussion - and your contributions elsewhere to the project!

I would be very excited to see pretty much everything you discussed here be implemented. Let's go through the points is some further detail:

The usefulness of an inventory management tool like InvenTree is directly correlated to its ease of use. Too tedious and folks wont use it.

100%

I'm aware of the TOS limitations of the various distributors (as discussed in #541 and #566), but there has got to be a better way. Take the digikey API for example, if the individual registers for a free API account, it appears they could use QR/barcode and Product retrieval API while adhering to the TOS.

I would like to see native integration (e.g. with Digikey) where all the user has to do is supply their API key (could be per-user or per-install).

What I would like to do is use my phone with the InvenTree app to scan bag barcodes.

I have long considered an integrated app to be a key feature for InvenTree - I have started working on this - https://github.com/inventree/inventree-app - but I have little (to no) experience working with Dart and also limited time.

I would love to see the app advanced. One of the key motivating factors would be to use a barcode scanner, with generated QR codes stuck on each inventory item (e.g. tubs / bags / etc).

These activities would take place in a "Parts Cache" tab in InvenTree

I really like this idea, I hadn't thought of having a "temporary working area". It could be quarantined to the current user, so e.g. the person in charge of receiving goods has a list of parts they still have to "deal with".

Part Actions

Everything here sounds really nice. It would probably require some fairly complex "business logic" on the back-end to make the user experience pleasant and simple. But, that's why we're here, right?

I have a fairly streamlined digikey API workflow in python for scanning barcodes, retrieving part/invoice/pricing info, and populating a spreadsheet (android).

Interesting - I think it sounds like there is a lot of functional overlap there!

This would allow laptops, desktops with webcams, and phones to all act as input sources.

Allowing the use of webcams would also assist with - #623 . I currently have no idea how to do this, it would be a super handy feature though!

My Thoughts

Pulling together the thoughts above, there are a few key features I think would be amazing to have:

Webcam Integration

A framework for accepting images from a webcam / connected camera would be great. Useful for uploading part images, or scanning barcodes.

App Integration

The integrated app would bring Inventree usability to everyone's pocket. Even just the ability to quickly check-out parts as you use them.

  • Use device's camera for barcode scanning
  • The REST API is already there (awaiting this app...)
  • Writing in dart (or similar?) for cross-platform support

Accepting Incoming Goods

This is a huge area where usability can be improved. I really like the idea of "scanning in" parts (either via a manufacturer barcode, or some other method) and placing them into a "staging area" - a virtual storage location where the user has to decide what to do with the parts. This "staging area" can be synced between the phone app and desktop interface - the "staged" parts are in a special database table waiting to be sorted and assigned.

The manufacturers barcode (Digikey / Mouser / etc) can be used to automagically work out how each incoming part maps to the database. A generic "manufacturer" framework can be implemented, so a new manufacturer style can be accommodated as needed.

TL;DR

The ideas you enumerated above are really nice, and I'd love to see them integrated!

@maholli
Copy link
Author

maholli commented Apr 4, 2020

@SchrodingersGat I'm glad to see your enthusiasm! Thank you for taking the time to discuss the ideas.

I spent a moment today streamlining my barcode scanning method:

  • Previously, I was using my phone to scan barcodes into a google sheet and then having a python script query the Digikey API for each line item.
  • Now it's running with a webcam and decoding the barcode locally. All in python. It turns out all the nice browser-based barcode scanner projects neglect the "data matrix" format that digikey and mouser use on their component bags. As a result, I cobbled the webcam and decoder functionality together in Python using OpenCV and pylibdmtx. Below is a quick diagram illustrating what I have working.

image

❓ Where I could use help is with the "business logic" mentioned in #677 (comment).

I will defer the SQL handling to those that are familiar with managing the flow of data into the db. An example of the resulting Digikey JSON I'm generating looks like this (some info redacted):

SEE UPDATE BELOW

Let me know if you're interested in playing with the webcam scanner / API query and I'll be more motivated to sanitize it and put it up on github. 🙂

Some items for folks to sink their teeth into:

  • Get something like zxing-js decoding data matrix QR codes to facilitate a cross-platform means of scanning component bags.
  • Build a nice "staging area" interface in InvenTree and brainstorm how to best check/add/manipulate these items in the db.

@maholli
Copy link
Author

maholli commented Apr 5, 2020

Barcode Update

It occurred to me that while we're going to all the trouble, we may as well query the product info for the scanned bag as well. Below is the full product info query for the scanned bag (some info redacted).

Things we could pre-populate with this information:

  • SKU, Manufacturer, PN, Description
  • Product description
  • Product image 🙂
  • Invoice ID (could be used to add line items to a PO)
  • Datasheet (link or maybe an option to cache locally)
  • For certain types of common parts (passives mainly?):
    • Type: resistor, capacitor, inductor, etc...
    • Package size: 0402, 0603, etc...
    • Value: 20pF
  • Price breaks 🙂
    Maybe it'd be best to give the InvenTree user the ability to select what product info gets displayed for certain product types?

Digikey vs Mouser vs LCSC API

  • Digikey API is much easier to work with since it takes barcodes (2D and 1D) directly.
  • Mouser will require some more work, but it's doable aside from the ToS bs:
    • Mouser directly states something I've been unable to find in Digikey's ToS:

      ... "you agree that you will not, and will not assist or enable others to: a. cache, record, pre-fetch, or otherwise store any portion of the Mouser Electronics Content"

This is pretty darn limiting. ONE potential workaround is populating our InvenTree data from purely the contents encoded in the barcode. There appears to always be quantity, MFPN, and invoice number...

  • LCSC barcodes are unusable for me. Anyone have a lead on these?

Digikey Product Info Query

Single JSON response

Typical response from a digikey QR scan:

{
   "AdditionalValueFee": 0,
   "AlternatePackaging": [
      {
         "DigiKeyPartNumber": "478-10464-2-ND",
         "Manufacturer": {
            "Parameter": "Manufacturer",
            "ParameterId": -1,
            "Value": "AVX Corporation",
            "ValueId": "478"
         },
         "ManufacturerPartNumber": "08052U200FAT2A",
         "ManufacturerPublicQuantity": 0,
         "MinimumOrderQuantity": 2000,
         "NonStock": false,
         "Packaging": {
            "Parameter": "Packaging",
            "ParameterId": 7,
            "Value": "Tape & Reel (TR)",
            "ValueId": "1"
         },
         "ProductDescription": "CAP CER 20PF 200V NP0 0805",
         "QuantityAvailable": 10000,
         "QuantityOnOrder": 0,
         "UnitPrice": 0.128
      },
      {
         "DigiKeyPartNumber": "478-10464-6-ND",
         "Manufacturer": {
            "Parameter": "Manufacturer",
            "ParameterId": -1,
            "Value": "AVX Corporation",
            "ValueId": "478"
         },
         "ManufacturerPartNumber": "08052U200FAT2A",
         "ManufacturerPublicQuantity": 12000,
         "MinimumOrderQuantity": 1,
         "NonStock": false,
         "Packaging": {
            "Parameter": "Packaging",
            "ParameterId": 7,
            "Value": "Digi-Reel\u00ae",
            "ValueId": "243"
         },
         "ProductDescription": "CAP CER 20PF 200V NP0 0805",
         "QuantityAvailable": 10902,
         "QuantityOnOrder": 0,
         "UnitPrice": 0.59
      }
   ],
   "AssociatedProducts": [],
   "Barcode": "[)>\u001e06\u001dP[CAP_01A]:ALT - C6,C7 - ALT\u001d1123123123123\u001dKPC11778128\u001d1123123123\u001d10K123123123\u001d11K1\u001d4123\u001dQ10\u001d11ZPICK\u001d1123123123\u001d1123123123\u001d20Z000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000",
   "DetailedDescription": "20pF \u00b11% 200V Ceramic Capacitor C0G, NP0 0805 (2012 Metric)",
   "DigiKeyPartNumber": "478-10464-1-ND",
   "ForUseWithProducts": [],
   "InvoiceId": 123123123,
   "KitContents": [],
   "Kits": [],
   "LeadStatus": "Lead Free",
   "LimitedTaxonomy": {
      "Children": [
         {
            "Children": [
               {
                  "Children": [],
                  "NewProductCount": 0,
                  "Parameter": "Categories",
                  "ParameterId": -8,
                  "ProductCount": 1,
                  "Value": "Ceramic Capacitors",
                  "ValueId": "60"
               }
            ],
            "NewProductCount": 0,
            "Parameter": "Categories",
            "ParameterId": -8,
            "ProductCount": 1,
            "Value": "Capacitors",
            "ValueId": "3"
         }
      ],
      "NewProductCount": 0,
      "Parameter": "Categories",
      "ParameterId": -8,
      "ProductCount": 1,
      "Value": "Out of Bounds",
      "ValueId": "0"
   },
   "Manufacturer": {
      "Parameter": "Manufacturer",
      "ParameterId": -1,
      "Value": "AVX Corporation",
      "ValueId": "478"
   },
   "ManufacturerLeadWeeks": "24 weeks",
   "ManufacturerName": "AVX CORPORATION (VA)",
   "ManufacturerPageUrl": "https://www.digikey.com/en/supplier-centers/a/avx",
   "ManufacturerPartNumber": "08052U200FAT2A",
   "ManufacturerPublicQuantity": 12000,
   "MatingProducts": [],
   "MediaLinks": [
      {
         "MediaType": "Datasheets                                                                                                                     ",
         "SmallPhoto": "",
         "Thumbnail": "",
         "Title": "Ceramic Chip Capacitor Catalog",
         "Url": "http://catalogs.avx.com/SurfaceMount.pdf"
      },
      {
         "MediaType": "Datasheets                                                                                                                     ",
         "SmallPhoto": "",
         "Thumbnail": "",
         "Title": "U Series, Ultra Low",
         "Url": "http://datasheets.avx.com/U-Series.pdf"
      },
      {
         "MediaType": "Product Photos                                                                                                                 ",
         "SmallPhoto": "http://media.digikey.com/Renders/AVX%20Renders/0805-(2012-Metric)-1,15mm_sml.jpg",
         "Thumbnail": "http://media.digikey.com/Renders/AVX%20Renders/0805-(2012-Metric)-1,15mm_tmb.jpg",
         "Title": "0805 (2012 Metric) 1,15mm",
         "Url": "http://media.digikey.com/Renders/AVX%20Renders/0805-(2012-Metric)-1,15mm.jpg"
      },
      {
         "MediaType": "Product Training Modules                                                                                                       ",
         "SmallPhoto": "",
         "Thumbnail": "",
         "Title": "Component Solutions for Smart Meter Applications",
         "Url": "https://www.digikey.com/en/ptm/a/avx-corporation/component-solutions-for-smart-meter-applications"
      },
      {
         "MediaType": "HTML Datasheet",
         "SmallPhoto": "",
         "Thumbnail": "",
         "Title": "Ceramic Chip Capacitor Catalog",
         "Url": "https://www.digikey.com/en/datasheets/avx-corporation/avx-corporation-surfacemount"
      },
      {
         "MediaType": "EDA / CAD Models",
         "SmallPhoto": "",
         "Thumbnail": "",
         "Title": "08052U200FAT2A by SnapEDA",
         "Url": "https://www.snapeda.com/parts/08052U200FAT2A/AVX/view-part/?ref=digikey"
      }
   ],
   "MinimumOrderQuantity": 1,
   "MyPricing": [],
   "NonStock": false,
   "Obsolete": false,
   "Packaging": {
      "Parameter": "Packaging",
      "ParameterId": 7,
      "Value": "Cut Tape (CT)",
      "ValueId": "2"
   },
   "Parameters": [
      {
         "Parameter": "Packaging",
         "ParameterId": 7,
         "Value": "Cut Tape (CT)",
         "ValueId": "2"
      },
      {
         "Parameter": "Part Status",
         "ParameterId": 1989,
         "Value": "Active",
         "ValueId": "0"
      },
      {
         "Parameter": "Capacitance",
         "ParameterId": 2049,
         "Value": "20pF",
         "ValueId": "u20pF"
      },
      {
         "Parameter": "Tolerance",
         "ParameterId": 3,
         "Value": "\u00b11%",
         "ValueId": "1131"
      },
      {
         "Parameter": "Voltage - Rated",
         "ParameterId": 14,
         "Value": "200V",
         "ValueId": "140848"
      },
      {
         "Parameter": "Temperature Coefficient",
         "ParameterId": 17,
         "Value": "C0G, NP0",
         "ValueId": "324183"
      },
      {
         "Parameter": "Operating Temperature",
         "ParameterId": 252,
         "Value": "-55\u00b0C ~ 125\u00b0C",
         "ValueId": "242918"
      },
      {
         "Parameter": "Features",
         "ParameterId": 5,
         "Value": "Ultra Low ESR",
         "ValueId": "417665"
      },
      {
         "Parameter": "Ratings",
         "ParameterId": 707,
         "Value": "-",
         "ValueId": "1"
      },
      {
         "Parameter": "Applications",
         "ParameterId": 405,
         "Value": "RF, Microwave, High Frequency",
         "ValueId": "396195"
      },
      {
         "Parameter": "Mounting Type",
         "ParameterId": 69,
         "Value": "Surface Mount, MLCC",
         "ValueId": "409416"
      },
      {
         "Parameter": "Package / Case",
         "ParameterId": 16,
         "Value": "0805 (2012 Metric)",
         "ValueId": "39329"
      },
      {
         "Parameter": "Size / Dimension",
         "ParameterId": 46,
         "Value": "0.079\" L x 0.049\" W (2.01mm x 1.25mm)",
         "ValueId": "8571"
      },
      {
         "Parameter": "Height - Seated (Max)",
         "ParameterId": 1500,
         "Value": "-",
         "ValueId": "1"
      },
      {
         "Parameter": "Thickness (Max)",
         "ParameterId": 1501,
         "Value": "0.045\" (1.15mm)",
         "ValueId": "6670"
      },
      {
         "Parameter": "Lead Spacing",
         "ParameterId": 508,
         "Value": "-",
         "ValueId": "1"
      },
      {
         "Parameter": "Lead Style",
         "ParameterId": 4,
         "Value": "-",
         "ValueId": "1"
      }
   ],
   "PrimaryDatasheet": "http://datasheets.avx.com/U-Series.pdf",
   "PrimaryPhoto": "http://media.digikey.com/Renders/AVX%20Renders/0805-(2012-Metric)-1,15mm.jpg",
   "PrimaryVideo": "",
   "ProductDescription": "CAP CER 20PF 200V NP0 0805",
   "ProductStatus": "Active",
   "ProductUrl": "https://www.digikey.com/product-detail/en/avx-corporation/08052U200FAT2A/478-10464-1-ND/6797372",
   "PurchaseOrder": "PC123123123",
   "Quantity": 10,
   "QuantityAvailable": 10902,
   "QuantityOnOrder": 0,
   "RoHSStatus": "ROHS3 Compliant",
   "RohsSubs": [],
   "SalesorderId": 123123123,
   "SearchLocaleUsed": {
      "Currency": "USD",
      "Language": "en",
      "ShipToCountry": "US",
      "Site": "US"
   },
   "Series": {
      "Parameter": "Series",
      "ParameterId": -5,
      "Value": "U",
      "ValueId": "1284"
   },
   "StandardPackage": 1,
   "StandardPricing": [
      {
         "BreakQuantity": 1,
         "TotalPrice": 0.59,
         "UnitPrice": 0.59
      },
      {
         "BreakQuantity": 10,
         "TotalPrice": 4.15,
         "UnitPrice": 0.415
      },
      {
         "BreakQuantity": 100,
         "TotalPrice": 27.32,
         "UnitPrice": 0.2732
      },
      {
         "BreakQuantity": 500,
         "TotalPrice": 101.2,
         "UnitPrice": 0.2024
      },
      {
         "BreakQuantity": 1000,
         "TotalPrice": 172.04,
         "UnitPrice": 0.17204
      }
   ],
   "SuggestedSubs": [],
   "TariffDescription": "",
   "UnitPrice": 0.59
}

@maholli
Copy link
Author

maholli commented Apr 10, 2020

API update + example

I went ahead and put my quickie webcam script up and then wrote a step-by-step for folks to get the example working with the new Digikey v3 API.

See https://github.com/maholli/barcode-scanner

https://raw.githubusercontent.com/maholli/barcode-scanner/master/images/demo.gif

@SchrodingersGat any progress making an InvenTree interface to accept these JSON responses?

@SchrodingersGat
Copy link
Member

@maholli that looks fantastic! Does the zxing javascript tool you mentioned above support this? It would be amazing to integrate this directly into the InvenTree web interface.

@SchrodingersGat
Copy link
Member

@SchrodingersGat any progress making an InvenTree interface to accept these JSON responses?

The JSON API is currently undergoing some heavy development mostly to support integration with an Android / iOS app

App Progress

image

image

image

the app currently supports scanning "InvenTree" style QR codes which allow the user to jump directly to a particular stock-location or stock-item. For example, say you have a box which contains a whole bunch of capacitors, you print a barcode that represents that box as a StockLocation. Scanning that barcode in the app then displays the information for that StockLocation.

This means that you do not have to lug a computer around etc, you have stock lookup and management right in your hand.

Third-Party Integration

There is obviously a lot of information available in the DigiKey / Mouser / LSCS / etc barcode. Where possible, these existing codes should be used to track StockItem objects rather than have to print out a separate barcode.

The information available could be mapped to the InvenTree data so that the digikey (or other manufacturer) barcode data is enough to uniquely represent a physical stock item. It also looks like there is other information available too, like Order ID. This could be used to mark items as "received" against a purchase order! The quantity is there too so partial-receiving could be supported.

Barcode Integration

The first step would be to work out a framework for generically handling manufacturer barcodes. I don't want to hard-code it for DigiKey and it doesn't work for Mouser, for example.

However we could have a generic "ManufacturerBarcode" class which then can be sub-classed for each manufacturer as the codes will need some custom management. Also some barcodes will have more data than others.

I would also like the data decoding to be handled on the server. This way the web interface and the app do not have to be "smart" and introspect the data, rather the server which has access to a lot more information works out what to do with the data.

The workflow would (basically) consist of the user-interface capturing the barcode data, and sending it to the server as a JSON object. Then the server works out if it can process it, and what to do with it.

Use Cases

Let's just focus on a DigiKey barcode.

Stock Scanning

Say you have a reel of capacitors and want to know how many parts are left in the reel. Just scan the DigiKey barcode, which will look up the StockItem in the database, and you'll see the current estimated quantity.

Incoming Goods

The barcode has order-id information, we could add a new field called "SupplierOrderID" or something to the PurchaseOrder, then an incoming item could be scanned and marked as "received" automagically

@maholli
Copy link
Author

maholli commented Apr 11, 2020

@SchrodingersGat holy cow, amazing progress! See my comments below for a few of the items you mention.

@maholli that looks fantastic! Does the zxing javascript tool you mentioned above support this? It would be amazing to integrate this directly into the InvenTree web interface.

Unfortunately it doesn't. In fact, there isn't a single open source barcode scanner with native browser support that reads data matrix (digikey & mouser) barcodes! How crazy.

RE: Barcode Integration

I don't want to hard-code it for DigiKey and it doesn't work for Mouser, for example.

I touch on this briefly in my comment above, but at the moment Mouser doesn't even have a public barcode API as far as I can tell. Nevertheless, I think linking all the data to the raw barcode value is the way to go, that way it's unique (this is in line with your third party integration comment).

State of barcodes

  1. Digikey: ✔ DK's API lets us fully query a barcode's product and invoice info
  2. Mouser: ➖ we'll need to regex the barcode and perform a Mouser API query of the extracted PN.
  3. LCSC: ❌ I'm unable to exact ANY usable info from the LCSC barcodes 😞. No API. For "internal use only"

I would also like the data decoding to be handled on the server.

This is excellent! In that case, is it that troublesome to use the python barcode decoder already employed in my digikey example?

Stock Scanning

OMG yes

@SchrodingersGat what else can I do to help? If decoding on the server with Python is too cumbersome, I can try and port data matrix handling to zxing (for example).

@rco3
Copy link

rco3 commented Apr 12, 2020

I signed up for GitHub, finally, so that I could come here and offer the little bits of work I’d done to extend the InvenTree Python API, and to ask about when some other features might become available… only to find Maholli laying down my own thoughts but far more eloquently, and with more progress on his own solutions. And that SG has reworked - is reworking - the JSON API, rendering obsolete my potential contributions but all in order to add a phone app to do all the things I dreamed about doing with InvenTree… Wow, you guys. Wow.

I’d really like to pitch in here if I can. I love the direction this is taking. I haven’t worked with Django before and my Python is rudimentary, so I have some coming up to speed on the coding side to do. However, I wanted to also contribute thoughts about the business logic and some use cases, if I may be so presumptuous:

It seems that there are really three types or aspects of Part in the database: Part, Supplier Part, and Stock Item. I understand (#680) that the Supplier Part is being transitioned to a Child of a Part, although I’m not sure what the progress thereof is - I love this idea, by the way. Does that mean that Stock Items are now going to be Children of Supplier Parts, which are themselves Children of Parts? In any case, the impetus to add a Part to the database usually comes from one end or the other of this chain, and only when needed. In other words, we add a Part because we have a design that requires that Part’s function, and then we add Supplier Parts to that Part, buy some, and those become a Stock Item. The other way is when we HAVE parts, and we need to keep track of them - how many, where, etc. In that case, we need to add a Stock Item, which requires a Supplier Part, which requires a Part. As Maholli points out, for both the Supplier Part and the Part we only create a new Part if we can’t find one already in existence. Could we have Stock Items associated with a Part without a Supplier Part?

I mention the two ends of the chain where we want to inject Parts; my current focus is a pair of part inventories, one at my day job and one at home. The former has a lot of recent parts, most with barcodes and most existing in a CAD system's database somewhere. The latter is by far the larger, far more varied, and many of the parts are bulk parts sorted into bags with handwritten labels. I need to be able to add these parts - both inventories! - in a batch-wise manner to my InvenTree database(s) as quickly and effortlessly as possible, or better yet get my 14-year-old to do it. It seems that the Part Template construct has the potential to support this, but that it either is incomplete or works differently than I understand. Operating under the assumption that it is simply incomplete, I’m currently envisioning a workflow that leverages the Part Template and Part Parameter constructs to simplify this immensely; see what you think. Key to this scheme are the ideas (perhaps pending implementation) that Part Templates have Children that are also Part Templates, and that Part Templates inherit Part Parameters and Keywords from their Parents and pass them on to their Instances.

I believe inheritance of Keywords and Parameters to multiple generations of Parent/Child Template relationships is key. Further, I think the ability to assign default values or lists to Parameters and select or confirm those values at Part Instance creation is highly useful, especially for commodity-type parts like resistors, machine screws, 74-series logic, Zener diodes, etc.

Example: ‘Resistor’ part has Parameters ‘Resistance’’Ohms’ and ‘Power’’Watts’. New Parts can be created from this Template with any additional Keywords or Parameters desired, or a Child Template can be created with additional Keywords, Parameters, and / or Parameter Defaults. e.g. ‘Metal Film 1% 1/4 W’ as a Child Template of ‘Resistor’ would add those terms as Keywords, would default ‘Power’ to 0.25 and would assign the list of standard E92 values to ‘Resistance’ as defaults. Another value of ‘Resistance’ could be specified at the time of Stock Item Instantiation if not available in the list. Of course, the Resistance value selected would also be added to the Keyword list, formatted in engineering notation (1k02 Ω, 33R2 Ω, etc.) for improved search ability. Standard checks for existing Parts apply, of course. Categories would be assigned to each Part Template - not inherited from Parents - and would be assigned to Parts created from that Template. Is there room to marry the Part Template and Category concepts more closely?

Part Template ‘Screw’ has only one Parameter, ‘Length’’in’, and one Keyword, ‘Screw’. However, Child Part Template ‘Machine, #10-32, SS316L Philips Panhead’ adds those terms as Keywords. When we create an Instance (‘Variant’?) of this Child Part Template, it has all of those keyword and the length we chose.

I don't think InvenTree needs to add those categories, of course - I expect each house to implement their own part taxonomy.

Of course, the IPN is another can of worms. If I create a set of House Part#s in my CAD system, I want to be able to use those same numbers as my IPNs. In this case, I’m likely to be adding a large number of parts in a batch via spreadsheet or similar. If I’m adding a tub of donated, bagged, hand-labeled components, I want to (optionally) generate a series of IPNs for my new Parts that follow a system or pattern. I might, for example, assign an ‘IPN Prefix’ of ‘RMFWJ’ to indicate that it is a ‘Resistor Metal Film 1% 1/4 W’, and then append a field containing the Engineering-Notation-Encoded resistance value followed by a very short serial number, to have something like ‘RMFWJ-1.02k-00’ automagically suggested as the new IPN. If I later added a new but slightly different part with a tighter TempCo or something, I might use the same Part Template but increment the serialized suffix and add a ‘±10ppm’ Keyword. Or I might create a new Child Part Template ‘Metal Film 1% 10 ppm 1/4 W’ that adds the Keyword by default.

I have a 2D barcode scanner that I’ve been using to parse DK’s DataMatrix packages; again, I’m clearly way behind here. My ideas about workflow dictate that the addition of a new Stock Item to the database should have the optional ability to print a QR code and some human-readable text; I use a Dymo LabelWriter 450, and would very much like to add a QR-code label to each of these new Stock Items when I create them. Stock Items added programmatically from scans of DK barcodes, on the other hand, shouldn't need a QR code printed as they have sufficient uniqueness as is.

Which reminds me - that webcam DataMatrix scanner is flat-out amazing. Seems like it wouldn't be that enormous an extension of that code to grab all of the barcodes on a bag label at the same time, if you knew where the various fields were located. I've got bags from Mouser, Arrow, NAC Semi, and Newark that all have multiple 1D codes with the info we typically need. We could basically treat that set of barcodes as one large unique Stock Item ID.

OK, I've probably gone on long enough. I sincerely hope I haven't stepped in where I'm not able to be truly helpful. Please let me know where you would like assistance directed, up to and including “leave us alone you idiot”. And thank you, Maholli and especially Schrodinger's Gat, for all the work you've done on this.

@maholli
Copy link
Author

maholli commented Apr 12, 2020

@rco3 I agree entirely with your comments, especially the part templates and IPN structures. I've been wondering how to best implement a custom IPN scheme myself and have yet to land on anything I feel is sufficient.

RE: webcam scanner

Glad you like it! I actually spent some more time on it last night and have it working for all Mouser DigiKey DataMatrix and 1D barcodes! 🎉 Here's a quick screenshot to illustrate your point about grabbing all the barcodes at once. Working well 👍. I'll update the repo shortly.

image

@maholli
Copy link
Author

maholli commented Apr 12, 2020

Barcode Update

Chugging along. Here's the current status of data extraction from the following suppliers:

  • Digikey: DK provides a fully functional barcode API for product and invoice info. 👍 good job DK!
    • ✔ 2D: specific API exists for invoice and product info queries. QR code is data matrix format.
      Always starts with [)>
    • ✔ 1D: specific API exists for invoice and product info queries. QR code is data matrix format.
  • Mouser: There's a part number query API, but no native barcode searching.
    • ✔ 2D: PN is extracted from DataMatrix (always starts with >[)>) and searched using Mouser's API
    • ➖ 1D: Script can read the barcodes, but currently has no way of telling which barcode value correlates to which property because there are seperate 1D barcodes for Cust PO, Line Items, Mouser P/N, MFG P/N, QTY, COO, and Invoice No.
  • LCSC: No API.
    • ➖ 2D: Some of my LCSC bags have QR barcodes (1 in 10 I'd guess). The QR code contains: productCode, orderNo, pickerNo,pickTime, and checkCode. So far all the tool can do is search LCSC for the PN but the user needs to navigate the page and extract the info. Need to write a javascript web scraper.
    • ❌ 1D: String ~10 characters in length. Can't extract anything useful from these.

This barcode scanning tangent project can be found at maholli/barcode-scanner

@SchrodingersGat
Copy link
Member

@rco3 there's a lot of good ideas there but probably too many varying ideas to tackle in this thread. Are you able to break some of the key points out into separate issues and we can discuss there? I won't have the bandwidth to deal with them all immediately, but at least we can make a start on them :)

@SchrodingersGat
Copy link
Member

@SchrodingersGat what else can I do to help? If decoding on the server with Python is too cumbersome, I can try and port data matrix handling to zxing (for example).

@maholli my preferred approach would be for the "client" device (web browser / barcode scanner / phone app / etc) to scan the barcode data (i.e. do the image processing component) and then send the decoded JSON data to the server, which can introspect the data and work out what to do with it.

This would probably mean using zxing for the browser interface. I believe that the barcode library I'm using for the app is based on zxing too.

Does zxing not currently support the data matrix codes? If you are familiar with the format it would be fantastic to add that feature and then we can work on supporting Digikey (and other suppliers) natively!

@SchrodingersGat
Copy link
Member

I have been considering how to integrate these external supplier codes natively. I think that #680 should be implemented first, so that all "Parts" can be associated with a supplier / manufacturer.

A question for you at this stage - does the digikey barcode data provide a unique identifier for an individual package (e.g. a single reel). If so, then we could add a "UUID" field to the StockItem class which we simply populate with the digikey ID. That way, can do a UUID lookup when we scan the digikey barcode and then locate the associated StockItem.

If there's no purpose-built UUID field we could do a hash of some other barcode data (order code / part code / date) to make a reasonably unique identifier.

@maholli
Copy link
Author

maholli commented Apr 12, 2020

A question for you at this stage - does the digikey barcode data provide a unique identifier for an individual package (e.g. a single reel).

@SchrodingersGat as far as I can tell, the barcode itself will always be unique. The downside is for digikey a barcode can be upwards of 356 characters (utf-8 encoded). This will likely get unwieldy, so we could hash it or something.

Does zxing not currently support the data matrix codes?

I originally dismissed it since folks were having trouble (like this), but I tested another online demo and got it to successfully decode a digikey datamatrix! Wahoo! (note I had to enable try-hard mode).

Decoding the barcode is only a single line in my example python script. Everything else is taking the barcode data and then doing various checks to find the supplier and then making the appropriate API call. So this should still be very helpful for the server to run once it has the received the decoded barcode from the client.

@SchrodingersGat
Copy link
Member

#722

@rco3
Copy link

rco3 commented Apr 14, 2020

SchrodingersGat,

I agree that it's a bit chaotic.

The Django webpage is... really cool. Adding one part is really easy. Adding a few parts is only moderately tedious. But I have large numbers of existing parts to add with: DigiKey Barcodes, Mouser/Arrow/etc barcodes, and NO barcodes., and I need to automate a lot of it. Hence the API.

I got to the point of adding Objects to the Python API for PartParameter and PartParameterTemplate, and a method for getting the Template for a given Parameter. But this recent work and discussion with you and Maholli may have had the effect of moving a lot of my process back to within the Django interface. Which I love.

So much of the point of the previous post was to suggest some ways to use Parameters, and ways to extend the current Template system which, like the Parameters system, appears to be a bit of a work in progress.

I shall enumerate more succinctly:

--Parameters are cool. I want to use them.

-- I like the idea of making SupplierParts Children of Parts. I wonder if that means that Stock Items are Children of Supplier Parts, or Children of Parts, or either?

--Part Templates should (my suggestion) carry Categories, Keywords, and Parameters which any new Parts created from that Template inherit.

-- Part Templates should be able to spawn Child Templates which inherit all of the Parent's Keywords and Parameters, add additional ones, and assign a new Category.

These are structural suggestions. I also would like to offer some process/interface suggestions:

-- Add Parts from Part Template: Pre-populate fields from the template wherever possible. After adding a part, offer to add another part with some fields (MFG, Vendor, maybe others) pre-populated from the previous part.

-- Offer various ways of pre-populating Parameter values:
-- Value assigned by Template
-- Value suggested by Template
-- Selected from a list of allowable values
-- Selected from a list of suggested values
-- entered directly

--Lists of values could be added in a table.
--Some tables could be included:
E92 resistor values,
74-Series logic numbers,
E24 Capacitance values,
SAE and Metric screw threads, and so on.
-- Otherwise, lists could be uploaded via API or accumulated via web interface.

Maholli and I are both interested in IPN generation. It's a topic upon which I have ruminated long, pondering the philosophy of the Part Number. Smart? Or Dumb? Random/serial? Or encoded and meaningful?

At the end of the day, I think that any scheme encoding enough information into part codes small enough to be usable requires significant memorization by the user to truly have any advantage over a random number looked up from a table. Plus, the effort of developing and maintaining such a scheme could be significant overhead for a small organization.

With that said, I believe that there could be some low-hanging fruit to be gathered by developing a basic IPN generator with provisions for:
-- a Prefix
-- Fields populated from Parameters
--- the List concept from above could be extended to have a Field Code associated with each Value. E.g., we could assign tolerance codes for resistors so that a Resistor whose Parameter is 1% (from a list) would have a Code of 'J' inserted into the automagical IPN, but a 0.5% would get a 'K', and so on.

I would be happy to separate some of these out into separate issues and post them, if you think there is merit. I fully recognize that there is a limit to your bandwidth.

@awulf
Copy link

awulf commented Dec 27, 2020

I'm loving the direction InvenTree is going, and have been running on a local server, and updating occasionally to see if it's in a state I'm able to use it more efficiently than a spread sheet.

The usefulness of an inventory management tool like InvenTree is directly correlated to its ease of use. Too tedious and folks wont use it.

This is the reason I haven't been able to use InvenTree yet for recording my electronic components at home. There's a lot of steps involved in entering in the basic values of a component like a capacitor with adding its paramaters, vs keeping a spreadsheet. Each parameter for each part requires 2 clicks. Compared to a tab press on the keyboard in a spread sheet.

Once the data is in there it seems easy enough to make use of.

For me personally I think the ability to import parts with their parameters in a CSV import be a good start to get the existing data in. Kinda like /admin/part/part/import/ does, but allow for extra columns that match parameter names for that category.

Might also be nice to have tabular view like /part/category/X/parametricbut being able to edit the values all at once in that view. Can make for quick data entry and corrections. Even better if you can add a row in that view.

@eeintech
Copy link
Contributor

eeintech commented Dec 28, 2020

There's a lot of steps involved in entering in the basic values of a component like a capacitor with adding its paramaters

I shared a lot of your opinion on this topic and Ki-nTree, a tool I'm working on, is automating the entry of parameters based on those fetched from Digi-Key. It relies on the InvenTree API to populate them.

You can also do the same with a quick python script that would take your spreadsheet as input and set your parts and parameters the way you'd like.

@SchrodingersGat
Copy link
Member

@awulf I agree that a tabular-style data entry system would really speed up the process. Probably at this stage the real blocker for implementation is bandwidth.

A tabular-style entry would require a lot of front-end code, although the back-end API is pretty fully featured already! It would require some investigation on how best to implement an editable table in the browser.

I think a table-style data entry system would be great for:

  • Part parameters
  • Bill of Materials
  • Adding items to orders

(And probably other use cases too!)

@rcludwick
Copy link
Contributor

@SchrodingersGat there's a couple of options I think I would consider maybe.

If you went with google spreadsheets, google would be doing most of the heavy lifting for you. The workflow would look something like this:

  1. Create the spread sheet in google sheets with the columns pre-made. https://googleapis.github.io/google-api-python-client/docs/dyn/sheets_v4.spreadsheets.html#create
  2. Let the user go to town on the input in the google api client
  3. Sync the data back in using a django q task you can kick off (since you have that now)

The second option is through an API.

  1. The user downloads a template CSV file.
  2. The user modifies it in the editor of their choice.
  3. The user saves it and drags it into the browser and and the browser posts the data to an API call that accepts CSV data.
    https://stackoverflow.com/questions/57483114/drf-upload-csv-file-then-iterate-over-each-row
  4. Rather than process everything in the API all at once, do some minimal validation to make sure the expected columns are there. Return a successful API return code, and save the CSV file on the server, say. Later, a django q task runs and imports the data properly.

@matmair
Copy link
Contributor

matmair commented Apr 20, 2021

@SchrodingersGat @rcludwick
There is add-in with functionality like that in bootstrap-table: X-Editable. There are some issues with the implementation but it is a good starting point.

@SchrodingersGat
Copy link
Member

Ooooh that does look very fancy. Might make a lot of existing form-based editing redundant if we can use in-line editables

@eeintech
Copy link
Contributor

Just FYI this project is quite dead...

image

@matmair
Copy link
Contributor

matmair commented Apr 26, 2021

@eeintech you are quite right about that, I did not see that. Quick search found something quite similar that could be implemented with the formatters from bootstrap-table.
It seems to be active (enough) and is written by a fullstack-dev that seems to keep everything simple and small. I will do a fast check on the weekend.

There is however no License, I opened a issue about that. And it is MIT!🙌

@SchrodingersGat SchrodingersGat added barcode Barcode scanning and integration enhancement This is an suggested enhancement or new feature labels Aug 12, 2021
@matmair
Copy link
Contributor

matmair commented Mar 4, 2022

@maholli the function you asked for should be possible to implement easily with the new plugin functionality.

@SchrodingersGat
Copy link
Member

Closing out this issue, it grew into quite the long-winded discussion and there's little in the way of specific actionable items here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
barcode Barcode scanning and integration enhancement This is an suggested enhancement or new feature
Projects
None yet
Development

No branches or pull requests

7 participants