Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

get_recourse and get_recourse_and_learn hang #2

Open
alexcappelletti opened this issue Aug 17, 2023 · 2 comments
Open

get_recourse and get_recourse_and_learn hang #2

alexcappelletti opened this issue Aug 17, 2023 · 2 comments

Comments

@alexcappelletti
Copy link
Collaborator

step to reproduce:

  1. build docker image
docker build -f .\Dockerfile --tag i3group/recourse-backend .
  1. run container
docker run --rm -it -p 5000:5000 i3group/recourse-backend 
  1. make a http request (post)
http://localhost:5000/get_recourse_v2

with parameters

{
    "features": {
        "adult": [
            {
                "name": "age",
                "value": 27
            },
            {
                "name": "capital_gain",
                "value": 0
            },
            {
                "name": "capital_loss",
                "value": 0
            },
            {
                "name": "education",
                "value": "Bachelors"
            },
            {
                "name": "hours_per_week",
                "value": 40
            },
            {
                "name": "marital_status",
                "value": "Never-married"
            },
            {
                "name": "native_country",
                "value": "United-States"
            },
            {
                "name": "occupation",
                "value": "Adm-clerical"
            },
            {
                "name": "race",
                "value": "White"
            },
            {
                "name": "relationship",
                "value": "Not-in-family"
            },
            {
                "name": "sex",
                "value": "Female"
            },
            {
                "name": "workclass",
                "value": "Private"
            }
        ],
        "lendingclub": [
            {
                "name": "acc_now_delinq",
                "value": 0
            },
            {
                "name": "acc_open_past_24mths",
                "value": 6
            },
            {
                "name": "all_util",
                "value": 40
            },
            {
                "name": "annual_inc",
                "value": 13000
            },
            {
                "name": "application_type",
                "value": "Joint App"
            },
            {
                "name": "avg_cur_bal",
                "value": 1761
            },
            {
                "name": "bc_open_to_buy",
                "value": 1596
            },
            {
                "name": "bc_util",
                "value": 82.8
            },
            {
                "name": "chargeoff_within_12_mths",
                "value": 0
            },
            {
                "name": "collections_12_mths_ex_med",
                "value": 0
            },
            {
                "name": "debt_settlement_flag",
                "value": "N"
            },
            {
                "name": "delinq_2yrs",
                "value": 0
            },
            {
                "name": "delinq_amnt",
                "value": 0
            },
            {
                "name": "dti",
                "value": 82.36
            },
            {
                "name": "emp_length",
                "value": "1 year"
            },
            {
                "name": "fico_range_high",
                "value": 734
            },
            {
                "name": "fico_range_low",
                "value": 730
            },
            {
                "name": "grade",
                "value": "E"
            },
            {
                "name": "hardship_flag",
                "value": "N"
            },
            {
                "name": "home_ownership",
                "value": "RENT"
            },
            {
                "name": "il_util",
                "value": 57
            },
            {
                "name": "initial_list_status",
                "value": "f"
            },
            {
                "name": "inq_fi",
                "value": 1
            },
            {
                "name": "inq_last_12m",
                "value": 2
            },
            {
                "name": "inq_last_6mths",
                "value": 2
            },
            {
                "name": "installment",
                "value": 809.02
            },
            {
                "name": "int_rate",
                "value": 26.3
            },
            {
                "name": "last_fico_range_high",
                "value": 664
            },
            {
                "name": "last_fico_range_low",
                "value": 660
            },
            {
                "name": "loan_amnt",
                "value": 20000
            },
            {
                "name": "max_bal_bc",
                "value": 4617
            },
            {
                "name": "mo_sin_old_il_acct",
                "value": 25
            },
            {
                "name": "mo_sin_old_rev_tl_op",
                "value": 91
            },
            {
                "name": "mo_sin_rcnt_rev_tl_op",
                "value": 20
            },
            {
                "name": "mo_sin_rcnt_tl",
                "value": 20
            },
            {
                "name": "mort_acc",
                "value": 0
            },
            {
                "name": "mths_since_last_delinq",
                "value": 18
            },
            {
                "name": "mths_since_last_major_derog",
                "value": 30.5
            },
            {
                "name": "mths_since_rcnt_il",
                "value": 20
            },
            {
                "name": "mths_since_recent_bc",
                "value": 35
            },
            {
                "name": "mths_since_recent_inq",
                "value": 0
            },
            {
                "name": "mths_since_recent_revol_delinq",
                "value": 9
            },
            {
                "name": "num_accts_ever_120_pd",
                "value": 0
            },
            {
                "name": "num_actv_bc_tl",
                "value": 3
            },
            {
                "name": "num_actv_rev_tl",
                "value": 5
            },
            {
                "name": "num_bc_sats",
                "value": 4
            },
            {
                "name": "num_bc_tl",
                "value": 4
            },
            {
                "name": "num_il_tl",
                "value": 3
            },
            {
                "name": "num_op_rev_tl",
                "value": 11
            },
            {
                "name": "num_rev_accts",
                "value": 13
            },
            {
                "name": "num_rev_tl_bal_gt_0",
                "value": 5
            },
            {
                "name": "num_sats",
                "value": 13
            },
            {
                "name": "num_tl_120dpd_2m",
                "value": 0
            },
            {
                "name": "num_tl_30dpd",
                "value": 0
            },
            {
                "name": "num_tl_90g_dpd_24m",
                "value": 0
            },
            {
                "name": "num_tl_op_past_12m",
                "value": 0
            },
            {
                "name": "open_acc",
                "value": 13
            },
            {
                "name": "open_acc_6m",
                "value": 0
            },
            {
                "name": "open_act_il",
                "value": 2
            },
            {
                "name": "open_il_12m",
                "value": 0
            },
            {
                "name": "open_il_24m",
                "value": 2
            },
            {
                "name": "open_rv_12m",
                "value": 0
            },
            {
                "name": "open_rv_24m",
                "value": 4
            },
            {
                "name": "pct_tl_nvr_dlq",
                "value": 100
            },
            {
                "name": "percent_bc_gt_75",
                "value": 75
            },
            {
                "name": "policy_code",
                "value": 1
            },
            {
                "name": "pub_rec",
                "value": 0
            },
            {
                "name": "pub_rec_bankruptcies",
                "value": 0
            },
            {
                "name": "purpose",
                "value": "debt_consolidation"
            },
            {
                "name": "pymnt_plan",
                "value": "n"
            },
            {
                "name": "revol_bal",
                "value": 9173
            },
            {
                "name": "revol_util",
                "value": 28.8
            },
            {
                "name": "sub_grade",
                "value": "E5"
            },
            {
                "name": "tax_liens",
                "value": 0
            },
            {
                "name": "term",
                "value": "36 months"
            },
            {
                "name": "tot_coll_amt",
                "value": 0
            },
            {
                "name": "tot_cur_bal",
                "value": 21129
            },
            {
                "name": "tot_hi_cred_lim",
                "value": 52900
            },
            {
                "name": "total_acc",
                "value": 16
            },
            {
                "name": "total_bal_ex_mort",
                "value": 21129
            },
            {
                "name": "total_bal_il",
                "value": 11956
            },
            {
                "name": "total_bc_limit",
                "value": 9300
            },
            {
                "name": "total_cu_tl",
                "value": 3
            },
            {
                "name": "total_il_high_credit_limit",
                "value": 21000
            },
            {
                "name": "verification_status",
                "value": "Verified"
            }
        ]
    },
    "preferences": {}
}
@geektoni
Copy link
Owner

geektoni commented Aug 21, 2023

I had a look at this, and it seems related to how uwsgi deals with threads and python.

Basically, the app is loaded in a pre-forking mode, in which the workers are just "forked" from the main process. Such processes should share all the objects of the main process, but sometimes there are issues with it. In our case, the predict() method of our pytorch models hangs, thus blocking the entire app

If we run uwsgi --ini uwsgi.ini --lazy-apps the problem disappears, at the cost of using much more RAM (each process will have its own copy of the ML models).

I did not find any clever way to fix it nicely, since I myself have no idea where the issue could lie specifically.

@alexcappelletti
Copy link
Collaborator Author

alexcappelletti commented Aug 22, 2023 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants