Skip to content

model-v3

Choose a tag to compare

View all tags
@AtulACleaver AtulACleaver released this 04 Oct 04:16
model-v3
3d275f3

Training commit SHA: bf4e0be

data-v3 Asset Hashes (SHA256):

  • baskets.parquet: a2dfd96cae05f12a32bc09f85ac4c32d1830e7e46681d98af8ebd7165c51b0bb
  • dataset.parquet: f6a06ba428981d3a0298f611b92706306644f8e36bdec03f07e607e158142752
  • skill_vocab.json: d749eb5d5044fea1e366972fba1bf94351c822dd580ac66a0e711a6f8f92dece
  • skills_autocomplete.json: aa45b9dff320373682a87c49bdfeee87ffabe181f13558305c78611c36ccf036

Model Selection:
We chose Logistic Regression because LightGBM's lead was under 3 points, favoring simplicity and interpretability.

Test Metrics:

  • Macro-F1: 0.742
  • Accuracy: 0.746 (Majority baseline: 0.177)
  • Top-3 Accuracy: 0.925

Note: The model is consistently overconfident in the upper probability bins (e.g., predicting 0.976 when the true probability is 0.915, or predicting 0.854 when true is 0.724).

Band Cutoffs (p75/p50 percentiles on val set):

  • Backend: Ready >=0.25, Close >=0.15
  • Data / BI Analyst: Ready >=0.20, Close >=0.10
  • Data Science / ML: Ready >=0.20, Close >=0.10
  • DevOps / Cloud: Ready >=0.15, Close >=0.10
  • ERP / Enterprise: Ready >=0.10, Close >=0.10
  • Frontend: Ready >=0.20, Close >=0.15
  • Full Stack: Ready >=0.25, Close >=0.15
  • Mobile: Ready >=0.20, Close >=0.15
  • QA / Test: Ready >=0.15, Close >=0.10
  • Support / IT Ops: Ready >=0.11, Close >=0.05
    (Probability cutoffs remain: Ready >=0.6, Close >=0.3)

Environment:

  • Python: 3.11
  • Packages: numpy 2.4.6, scipy 1.17.1, scikit-learn 1.9.1, pandas 3.0.6, pyarrow 25.0.1, rapidfuzz 3.14.6, lightgbm 4.7.0, mlxtend 0.25.0

Asset Hashes (SHA256):

  • classifier.pkl: 9eb0a756879d1491ed2e9f9acb6f71f43ffdf452e2ed725b1795bd2bef5a7a8b
  • label_encoder.pkl: b267b408e9b0b9be719c9cfde23858ae9f94780c0a2ef584e61f1252ea31454e
  • metrics.json: 05cfb86195be431c1699b1cd5b30d3ea31d405e184beded65f9430793ab2dd13
  • role_profiles.json: 72d6a8d886b1d173adf7dd27c8448e3b164831b230cfafb52ccdb88616e5b6cf
  • rules.parquet: cd2842f8495b16a9f58d2dad6d68925499fb362696f48c73817331bb0f256cfe
  • skill_vocab.json: d749eb5d5044fea1e366972fba1bf94351c822dd580ac66a0e711a6f8f92dece
  • skills_autocomplete.json: aa45b9dff320373682a87c49bdfeee87ffabe181f13558305c78611c36ccf036
    (Byte-identical copies of skill_vocab.json and skills_autocomplete.json are included so one download fills ARTIFACTS_DIR)
Assets 8
Loading