ML Custom Data Product for PII CSV

[Shakthieshwari]

Hi,

As part of next release-5.2, We are developing a PII Feature for Manage Learn Use-Case, as part of this we have requirement for CSV Exhaust.
We are planning to create a new ML custom data product for PII CSV from Cassandra . Below are the details :-

1. Story Link :- https://project-sunbird.atlassian.net/browse/LR-285
2. Design Doc :- https://project-sunbird.atlassian.net/l/cp/Aci0U7pY

Approach is detailed out in the design doc, Please check.

@SanthoshVasabhaktula @reshmi-nair @rhwarrier @kameshbhr Can you please let us know your availability this week to review? and also let us know the plan for 5.2 release in Lern BB?

Cc- @aishwaryashikshalokam @Prateek-slokam @aks30 @vijiurs @kiranharidas187

[rhwarrier]

@Shakthieshwari @Prateek-slokam if we were to draw a parallel with existing Diksha capabilities, this would be similar to the Course mentor getting access to user details of those people who have given consent? In this context, this becomes a Program, and the details are of users who are part of the Program?
Isn't it possible to reuse the same report that we use for courses, in the context of Programs?
@SanthoshVasabhaktula please provide your inputs.

@reshmi-nair Given consent reports are part of Lern, this would also be so?

[aks30]

@reshmi-nair can you share information about this? Consent history is something we need.

@reshmi-nair the question from @Shakthieshwari here is regarding the story requirement listed here - https://project-sunbird.atlassian.net/browse/ED-599, which talks about the user can give consent and revoke and again consent for the same resource, in which case we would want to know the history and track of the consent given by the user for a particular resource, so if that is possible in the current implementation or not

[reshmi-nair]

@aks30 No we don't have consent history for same org or collection. If a consent is revoked for one org/collection and gave consent again then same record gets updated. The record will have created date and last updated date.

[SanthoshVasabhaktula]

@reshmi-nair - Then it is not as per design. Consent history should always be stored. Are you creating an audit event for all consent changes? Please create a bug and take it up in the next release.

@rhwarrier

[kiranharidas187]

@reshmi-nair @SanthoshVasabhaktula We are trying to integrate user consent API as confirmed earlier
Payload
{ "request": { "consent": { "status": "REVOKED", "userId": "b3649713-cfce-4a89-83dc-7b19730a6ee8", "consumerId": "0126796199493140480", "objectId": "6319a4d53c40dd000978dacb", "objectType": "Program" } } }

Response
{ "id": "api.user.consent.update", "ver": "v1", "ts": "2023-01-27 07:44:39:256+0000", "params": { "resmsgid": "ea250a438f4e487b50926ec574862d37", "msgid": "ea250a438f4e487b50926ec574862d37", "err": "UOS_UCNSNTUPD0019", "status": "FAILED", "errmsg": "Please provide valid organisationid." }, "responseCode": "CLIENT_ERROR", "result": {} }

The consumerId in the payload we populated with users rootOrg channeled, but still we are getting the error. Can you let us know what should be the value of consumerId key?

Actually we don't require any orgId for our use cases, we are just adding that as consumerId key is mandatory for the API call. FYI

[reshmi-nair]

@SanthoshVasabhaktula Audit events are generated during consent changes.

@kiranharidas187 We userid, consumerid and objectid is mandatory in consent. You can pass the objectid 's( program 's ) rootOrgid as consumerid.

[Shakthieshwari]

Lastest Comment by @SanthoshVasabhaktula in this #54 forum, Continuing the conversation here

@Shakthieshwari @aishwaryashikshalokam - I am not sure I understand clearly what is meant by filtering. Is it that the job will be submitted with a filter? If so, filters can be added during exhaust job submission and the data product can process it.

Answer :- yes @SanthoshVasabhaktula, Filters will be added to the Job Request and query will be constructed based on that in the data product and process it .

But if it is to access via the UI, I am not sure how a password protected file can be displayed in UI. Can anyone elaborate on this approach?

Answer :- We are not displaying the password protected file on the UI, we are giving option for Program Manager to download the password protected csv file from the Program Dashboard .

Attaching below the architectural design of Program Dashboard for better understanding ... Currently on the Program Dashboard Datasets today, we are downloading the password protected csv file

Another question - How is the consent information captured and recorded? Is the current consent API going to be used for it?

Answer :- @aks30 @kiranharidas187 @aishwaryashikshalokam @Prateek-slokam request you to please respond to this question

It is fine to capture user details during joining the program and storing it in Cassandra or any other transactional DB. However, the PII data should be encrypted before storing.

Answer :- PII Data - UserName,Email id and Mobile Number will not be stored in Cassandra or any other transactional DB, during the CSV Generation i,e in the data product with the help of user id we will be querying the redis cache and get the details. Only the Entity Level Data Like State,District,Block,Cluster,School,Organisation,User Type,User Sub Type will be stored in Cassandra or the transactional DB
@aks30 @kiranharidas187 Please provide your input as well

In addition, if it is a very specific use case targeted to be only used in DIK, create the job in DIK repository but not SB-LERN.

No, It is not specific use case targeted to only DIK

@rhwarrier

[SanthoshVasabhaktula]

@Shakthieshwari - Can you articulate why it is not DIK specific? or how the solution is generalised to support various use-cases beyond DIK? If you do not have the answer - please create the job in DIK only for now.

[Shakthieshwari]

@SanthoshVasabhaktula will discuss in our call tomorrow. Thanks

[Shakthieshwari]

@SanthoshVasabhaktula As we discussed today in the call, this query is sorted out.

[Shakthieshwari]

@SanthoshVasabhaktula Can we please get on a call for 30-45mins to review our design either tomorrow or on monday ? please let us know your availability ?

Thanks

[SanthoshVasabhaktula]

@Shakthieshwari. Sure - Tomorrow (17th jan) second half after 3 PM works fine. Please ensure the architect from your end also joins the call.

[Shakthieshwari]

@SanthoshVasabhaktula Thanks. I have shared the invite for tomorrow at 3pm. Please do join

[Shakthieshwari]

@rhwarrier As we discussed, you will confirm us that in which Building Block does this Data product will go ?

I updated this data product in the component list as well...

[rhwarrier]

Let's close on this in the meeting tomorrow. The current SB job that pulls user PII (for course mentors/ org admins) is in Lern - not because it is a good fit there, but because it does't fit in anyplace else.

[Shakthieshwari]

@SanthoshVasabhaktula As per our discussion, I have updated the doc. https://project-sunbird.atlassian.net/wiki/spaces/AN/pages/3266675098/Cassandra+Approach+for+PII+data

Please review and provide your approval for the same ...

@rhwarrier can you please provide your approval? if we can use Sunbird-Lern Building Block for this data-product, as we could not discuss this in our today's call.

Thanks

[Shakthieshwari]

@SanthoshVasabhaktula can you please review this design doc and provide your approval at the earliest ?

We need to get started with the implementation, because PII is very high priority for us. Please do the needful at the earliest...

Thanks

[SanthoshVasabhaktula]

@Shakthieshwari - Most of the design is fine. However there are following concerns:

1. I do not see the need to store block, school, cluster name & code in the program_enrolment table. It can be joined within the job
2. I am not comfortable with PII in plain text within the csv even though the zip is password protected. The PII has to be encrypted within the file.

You can start of on the implementation, but we will revert back on the PII encryption within the file and it may need to be implemented before the code is merged.

@rhwarrier @madhucr

[Shakthieshwari]

Thanks @SanthoshVasabhaktula .

Please find the response below for the concerns you have mentioned :-

1. I do not see the need to store block, school, cluster name & code in the program_enrolment table. It can be joined within the job

**Answer :-** As per the requirement, State,District,Block,Cluster,User Type,User Sub Type,Organisation of the user to be captured at the time of joining the program, because user can update the profile at any time after the joining the program. Hence we decided to capture the entity details at the time of joining the program and thus storing it in the **program_enrolment** cassandra table . This is already updated in the confluence doc.

Please find below some examples to illustrate this scenario :-

**Ex :- At the time of joining the program, these are the user profile details**
        State = Andhra Pradesh	
        District = Visakhapatnam
        Block = Pedagantayada
        School = Zphs Nadupuru
        ......
        
        **After Few days of joining the program, user update their profile details to** 
        State = Andhra Pradesh	
        District = Visakhapatnam
        Block = Chinagadila
        School = Zphs Chandram Palem
        ......
        
        In the User download CSV, they should see these entity details at the time of joining the program and not an updated profile. 
        CSV Data :-
        State = Andhra Pradesh	
        District = Visakhapatnam
        Block = Pedagantayada
        School = Zphs Nadupuru
        ......

2. I am not comfortable with PII in plain text within the csv even though the zip is password protected. The PII has to be encrypted within the file.

**Answer :-**  Currently Course User CSV has PII Data in Human Readable Format/Plain text only, similar to this we will also create the same logic for our ML PII CSV. 

If PII Data has to be encrypted within the file, then i think this could be a change across all CSV's either in Course User CSV/ ML PII CSV ....

We will get the implementation started for now and will be waiting for your reply back on this point( PII encryption within the file).

Thanks

[rhwarrier]

@Shakthieshwari The data product can be part of Lern. You can initiate work once you close on the DC design review. Thanks .
@reshmi-nair we'll have to help the team with inputs - around documentation, test cases and microsite updates that will be n needed as part of the contribution.

[Shakthieshwari]

Thanks @rhwarrier

[Shakthieshwari]

@reshmi-nair @SanthoshVasabhaktula Can you please let us know in which BB/Git repo this PII Python Flink Job with Kubernetes will be moved ?

[Shakthieshwari]

@snehangsude please take this forward

[reshmi-nair]

@Shakthieshwari Lern flink jobs are in data-pipeline. You can create one module for ML here and add the flink job.
cc: @SanthoshVasabhaktula @rhwarrier

[snehangsude]

@kumarks1122
Could you please help clarify a few points?

• Could you help in how would Python dependencies (libraires) be installed in the data pipeline? Is there a structure that is being followed - which we need to maintain?
• How would we be linking them with the Flink job - as in which folder contains information/structure/code regarding to this from the Github ?
• How are we storing the configuration (private variables) in the repo?

Thanks.

CC: @reshmi-nair @Shakthieshwari

[kumarks1122]

@snehangsude : Kindly find the details in below.

• Could you help in how would Python dependencies (libraires) be installed in the data pipeline? Is there a structure that is being followed - which we need to maintain?
  Currently flink jobs are written in scala. Please follow https://nightlies.apache.org/flink/flink-docs-master/docs/dev/python/dependency_management/ for adding dependencies to the python application.
• How would we be linking them with the Flink job - as in which folder contains information/structure/code regarding to this from the Github ?
  Flink jobs are linked by kafka topics. But for deploying in same flink cluster where scala jobs are running, follow the commands in the doc https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/cli/#submitting-pyflink-jobs
• How are we storing the configuration (private variables) in the repo?
  Private values are stored in private repo and variables are interpolated with values which is in ansible and kubernetes folder.

[Ashwiniev95]

Hi @kumarks1122

We are working on Scala code for creating a new Flink job in data-pipeline,
Kafka is used as a source to consume the data, having this case we need to define events schema for transformation. Since Event class is defined in dp-core module, We trying to reuse by extending that class in our module. But EventsPath class that is defined dp-core, which is specific to Telemetry data.

So can we use this in our module ?

-CC @Shakthieshwari

[SanthoshVasabhaktula]

@Ashwiniev95 - Use String or Map source/sink if the event structure doesn't follow telemetry spec. Within your process function you can re-map it to POJO or case class if needed

[Shakthieshwari]

Hi @kumarks1122

We have started implementation of data product for PII similar to UserInfoExhaust Job, I am facing 1 issue :-

Trying to save the dataframe into local disk, calling this function

val files = reportDf1.saveToBlobStore(storageConfig, fileFormat, filePath, Option(Map("header" -> "true")), None)
But I am not able to see the csv file stored into local disk .

Please find my analysis below i have done to debug this issue

Values passed to this functions 👎

storageConfig = StorageConfig("local",None,"reports/",None,None)
fileFormat = csv
filePath = program-user-exhaust/37564AN8F134RR7532F125651B51S17D_20230227.

Dataframe :-

+--------------------+--------------------------+------------------------------+-------------------------+----------------+---------------------+--------------------+--------------------+--------------+---------+--------+----------+-----------+--------------------+-------------+-------------+--------------------+
| User UUID|User Name(On user consent)|Mobile number(On user consent)|Email ID(On user consent)|Consent Provided|Consent Provided Date| Program Name| Program ID| State| District| Block| Cluster| School Id| School Name| Usertype| Usersubtype| Org Name|
+--------------------+--------------------------+------------------------------+-------------------------+----------------+---------------------+--------------------+--------------------+--------------+---------+--------+----------+-----------+--------------------+-------------+-------------+--------------------+
|3fa8dd7a-6426-48f...| | | | false| |Prerak Head Teach...|PGM-Prerak-Head-T...| Uttar Pradesh| AGRA|ACHHNERA| |09150206302|JHS NAGLA SADLE C...|administrator| hm|dikshapreprodcust...|
|19569316-7b66-4b9...| Test user 1 | 8618649658| admin@gmail.com| true| 01/01/2023|Prerak Head Teach...|PGM-Prerak-Head-T...| Uttar Pradesh| AMETHI| AMETHI| |09730200601| PS UMAPUR GANAPATTI|administrator|diet_pricipal|dikshapreprodcust...|
|489be8be-f6ab-482...| teststag44 | | teststag4@yopmail...| true| 07/11/2022|Prerak Head Teach...|PGM-Prerak-Head-T...|Andhra Pradesh|ANANTAPUR| AGALI|ZPHS AGALI|28226200816|SMT PRAMEELAMMA A...|administrator| deo|dikshapreprodcust...|
+--------------------+--------------------------+------------------------------+-------------------------+----------------+---------------------+--------------------+--------------------+--------------+---------+--------+----------+-----------+--------------------+-------------+-------------+--------------------+

saveToBlobStore function is present in sunbird-analytics-core git repo, I debugged each line by line by putting debugger/print statements. But could not figure out any issues.

And also I tried to execute OnDemandDruidExhaustJob Data Product, this also calls the same saveToBlobStore Function only present in analytic-core, But here its saves data into local disk.

Attaching the Screenshot of Intellij Below for your reference ...

Below is the Model Config :-

{
  "search": {
    "type": "none"
  },
  "model": "org.sunbird.ml.exhaust.ProgramUserInfoExhaustJob",
  "modelParams": {
    "store": "local",
    "mode": "OnDemand",
    "authorizedRoles": [
      "PROGRAM_MANAGER"
    ],
    "id": "ml-program-user-exhaust",
    "keyspace_name": "sunbird_programs",
    "table": [
      {
        "name": "program_enrollment",
        "columns": [
          "user_id",
          "program_name",
          "program_externalId",
          "state_name",
          "district_name",
          "block_name",
          "cluster_name",
          "school_code",
          "school_name",
          "user_type",
          "user_sub_type",
          "organisation_name",
          "pii_consent_required"
        ]
      },
      {
        "name": "user_consent",
        "columns": [
          "user_id",
          "status",
          "last_updated_on"
        ]
      },
      {
        "name": "user",
        "columns": [
          "userid",
          "firstname",
          "lastname",
          "email",
          "phone",
          "username"
        ],
        "encrypted_columns": [
          "email",
          "phone"
        ],
        "final_columns": [
          "email",
          "phone",
          "username"
        ]
      }
    ],
    "label_mapping": {
      "user_id": "User UUID",
      "username": "User Name(On user consent)",
      "phone": "Mobile number(On user consent)",
      "email": "Email ID(On user consent)",
      "consentflag": "Consent Provided",
      "consentprovideddate": "Consent Provided Date",
      "program_name": "Program Name",
      "program_externalId": "Program ID",
      "state_name": "State",
      "district_name": "District",
      "block_name": "Block",
      "cluster_name": "Cluster",
      "school_code": "School Id",
      "school_name": "School Name",
      "user_type": "Usertype",
      "user_sub_type": "Usersubtype",
      "organisation_name": "Org Name"
    },
    "order_of_csv_column": [
      "User UUID",
      "User Name(On user consent)",
      "Mobile number(On user consent)",
      "Email ID(On user consent)",
      "Consent Provided",
      "Consent Provided Date",
      "Program Name",
      "Program ID",
      "State",
      "District",
      "Block",
      "Cluster",
      "School Id",
      "School Name",
      "Usertype",
      "Usersubtype",
      "Org Name"
    ],
    "sort": [
      "district_name",
      "block_name",
      "cluster_name",
      "school_name",
      "user_id"
    ],
    "sparkElasticsearchConnectionHost": "localhost",
    "sparkRedisConnectionHost": "localhost",
    "sparkUserDbRedisIndex": "0",
    "sparkCassandraConnectionHost": "localhost",
    "sparkUserDbRedisPort": 6381,
    "fromDate": "",
    "toDate": "",
    "key": "reports/",
    "format": "csv"
  },
  "output": [
    {
      "to": "file",
      "params": {
        "file": "reports/"
      }
    }
  ],
  "parallelization": 8,
  "appName": "Program UserInfo Exhaust"
}

Wondering what could be the issue in the PII Data Product and also there is no error, due to which unable to figure out this issue. Can you please help us out here ?

Thanks

[Shakthieshwari]

@kumarks1122 This issue is sorted.

[Shakthieshwari]

@reshmi-nair ML Data Product for PII is part of 5.2 Sunbird Lern BB release. As per the timeline, we will be raising a PR by 10th March. Once the PR is raised and merged, how is the deployment being followed ? where we will deployment happens?

Because till now we have not contributed anything on Lern BB, so we wanted to know the process of deployment and how the testing will happen forward ?

Can you please help us out here

Thanks

[reshmi-nair]

@Shakthieshwari Pls refer this PR for adding a new dataproduct in to lern. PR link
Add the job details in ansible/roles/lern-data-products-deploy/templates/lern-model-config.j2 and ansible/roles/lern-data-products-deploy/templates/lern-run-job.j2

Also we need to add the job id into below config file in devops repo.
https://github.com/project-sunbird/sunbird-devops/blob/release-5.1.0/deploy/jenkins/jobs/Deploy/jobs/dev/jobs/Lern/jobs/LernAnalyticsReplayJobs/config.xml

If any private configurations pls add into sunbird-devops-private repo 'release-5.2.0-lern' branch
/ansible/inventory/dev/KnowledgePlatform/common.yml

[reshmi-nair]

@Shakthieshwari Private configuration means, the secret keys or IPs or other stuff that cannot be configured in public repo. Usually all those things are added to devops-private repo.

For deployment, the job id has to be configured in lern-model-config.j2 and lern-run-job.j2 files in Lern/data-products repo. Also the same thing needs to be configured in jenkins analytic replay jobs. I shared these details in above message.

Everything has to be updated in the release notes. If it is a daily job and need to be configured as cron job, pls mention those details also in the release notes.

These changes cannot go in release-5.2.0. DC has mandated that, all PII related reports have to be encrypted. Design work has started on this. Once design is approved, i will share that. Include those also and submit the PR.

[Shakthieshwari]

got it, thanks @reshmi-nair .

@reshmi-nair Can you please let us know the process for deployment, testing and if any technical document to be prepared for update in the micro-site ?

[Shakthieshwari]

@reshmi-nair where do we update the release notes ? If you please, share some link

In the Current Design PII CSV/Report is a encrypted file only and DC Approval by Santhosh V, was already discussed before we started the implementation. Now what other approval is required ?

Thanks

[reshmi-nair]

@Shakthieshwari Release notes link for 5.2.0 i will share next week. Currently the reports are not encrypted, only password protection is given to the zip files. Santhosh V has come up with some enhancements for encrypting the reports with PII info as part of design level security auditing. Currently this is in design phase.

[Shakthieshwari]

@reshmi-nair Got it. Thanks for the clarifications

[Ashwiniev95]

Hi @reshmi-nair

As part of 5.2 Sunbird Lern ML Data Product for PII is created, to support this with Cassandra table, ML Data Pipeline Flink job is also built.
We wanted to know the configurations details ,process of deployment and how the testing will happen for this Flink job.

Can you please help us on this

Thanks

[Shakthieshwari]

@reshmi-nair FYI, The changes whatever is mentioned above for both Flink Jobs and Data Product will be completed and PR will be updated by tomorrow EOD (30-03-2023)

[reshmi-nair]

@Shakthieshwari Is the PR ready? We are waiting for your update to close the 5.2.0 release. Also pls share the test scenarios and configurations and other details need to be added to release notes.
@rhwarrier

[Shakthieshwari]

@reshmi-nair As i informed yesterday, By today EOD PR will be ready for both Flink Job and Data Product. Since Data Product will be taken as part of Diksha 6.0 release, you can review and merge Flink PR. And also if possible please also review Data Product PR as well and provide your comments on it, so that it would be helpful and we will ready.

Do you want the test scenarios and configurations in a document ? As we are aware off how to add it in the release notes.

Thanks

[Ashwiniev95]

HI @reshmi-nair

As per the comments both Data-pipeline and data product changes and made and code is pushed to the git. Please find the Data-pipeline PR link Sunbird-Lern/data-pipeline#103

-CC @vijiurs

Thanks

[reshmi-nair]

@Ashwiniev95 Data-pipeline PR is merged. Pls share the testcases and configuration details for release notes

[Shakthieshwari]

@reshmi-nair ML PII- Program UserInfoExhaust Data Product Implementation is completed, Can you please review and merge the below PR ?
Sunbird-Lern/data-products#34

And also changes is done in devops repo as well, Please find the below PR
project-sunbird/sunbird-devops#3759

I have pinged devops to merge this PR as well.

Please do the needful at the earliest ....

Thanks

Cc- @vijiurs

[Shakthieshwari]

@reshmi-nair Can you please help merge the above PR for the data product ? So that we can proceed further with deployment and testing...

Thanks

[reshmi-nair]

@Shakthieshwari With respect to the design audit conducted, DC has mandated that, all PII related reports have to be encrypted before storing. As the PR that you have shared also is creating a report with PII data, this can be merged only if the file is encrypted. So this PR cannot be merged now.

On PII data encryption, design work is going on and this will be planned in release-5.3.0 of Lern.
cc: @rhwarrier @SanthoshVasabhaktula

[Shakthieshwari]

@reshmi-nair As we discussed, this is very high Priority for us, Can we please setup a call with DC by tomorrow quickly for 30mins ?
@rhwarrier @SanthoshVasabhaktula @reshmi-nair Can you please let me know your availability for call tomorrow ?

@aishwaryashikshalokam @Prateek-slokam Can you please explain about the Priority of this ?

Sunbird Lern :- Release Calendar :- https://lern.sunbird.org/use/release-calendar

[rhwarrier]

@Shakthieshwari I've set up some time tomorrow. Let's find a way to take this forward in a manner that works for all.

[Ashwiniev95]

Hi @reshmi-nair ,
As per our discussion,

I have raised the PR in sunbird-utils, Please find the link below
Sunbird-Lern/sunbird-utils#1149

I have also created the release-notes, Please find the link below
https://docs.google.com/document/d/1PIe6g9YeACwEXzLhS0Q0uuF8GACI5Ljt-wSWmfBQgBg/edit

-CC @vijiurs

Thanks!

[Shakthieshwari]

@reshmi-nair As per this https://docs.google.com/document/d/1Pq0oWdeSNWWw2mGQelXpGmj8b_NS5Q3ndt0folpj2r4/edit# Design Doc, Can you please help us resolve the below mentioned queries :-

1. What is the Cassandra keyspace name, table name, schema used for UserOrg Tenant Perference API's ?
2. If Exception is thrown based on Security Levels L1,L2,L3,L4, what is the use of this exception ? how is the user notified about the exception in the Portal ?
3. If the decryption tool is a python script, what if the end user is non-technical ? how will this tool help them out ?

And also once, Code is pushed and PR is merged for UserInfoExhaust Data Product in Sunbird-Lern BB from your end, Please let us know. We will pull in our local and do the changes for our PII Data Product similar to UserInfoExhaust

Thanks

[Shakthieshwari]

@reshmi-nair Please let us know once the PR is merged, with the PR link you shared, we can only go through the logic for security feature. But UserInfoExhaust Job Changes is not part of the PR you shared, hence we cannot start the implementation. Hence please share us the PR Link of UserInfoExhaust Job Changes along with Security Features Added, then we can start the implementation....

For now, we will go through the code...

[Shakthieshwari]

@reshmi-nair Any update on this Please ? Please provide an ETA on this ? And also let us know the 5.3 Lern Release Dev End Date, because we would need at-least a week to implement.

[reshmi-nair]

@Shakthieshwari All the changes are in the PR shared with you. Changes are in common files : BaseCollectionExhaustJob.scala, OnDemandExhaustJob.scala. Not in specific job files.

[Shakthieshwari]

@reshmi-nair Any Update by when this Sunbird-Lern/data-products#38 PR will be merged ? 5.3 Lern Development is already ended, but still this PR is not merged.

We would need atleast 4 days to push our new PII Data Product, Please let us know the plan

[Shakthieshwari]

@reshmi-nair I think this PR is getting updated still, I could see lot of changes in this PR now.Let me know by when this PR will be merged ? Only then we can pull the changes and add our code to it

[rhwarrier]

@reshmi-nair can you pls let the SL team know?

[rhwarrier]

@AmiableAnil pls take a look

[Shakthieshwari]

@AmiableAnil I have replied to your PR comments, Please request you to revert back to me and merge this PR as early as possible
Cc- @aks30 @vijiurs

[Shakthieshwari]

@AmiableAnil As we discussed, We are facing below issues in testing our data product on dev lern env :-

1. Update Preference API is not working, Getting 403 Forbidden. As you requested, I have shared the curl command in the slack.
2. PR Raised for the Data Product is still not merged because of Circle CI Failing, In my local Build is success. You told you will check with reshmi and get back to me, But still their is no any update on it
   https://github.com/Sunbird-Lern/data-products/pull/40/files
3. Spark machine and Rancher access we got, but not sure how to use it and where to find out the logs, Request you to please help us
4. Tried to submit the job request with default(org) as the value for the key x-channel-id. But API is failing with 500 Internal Server Error. Shared you the curl in slack

Please request you to check all these issues and get back to me at the earliest .... As these issues are blocking our testing

Cc- @aks30 @vijiurs

[Shakthieshwari]

@AmiableAnil And also, Data Product is generating CSV Today. But the Link is not working, giving me BlobNotFound Error. Attaching the screenshot below
Tag Id :- program_607d3000aba99f53949dda45:01250894314817129555
Link :- https://devbbprivate.blob.core.windows.net/reports/ml_reports//program-user-exhaust/39C3C8210212C566DD8CA096FA587DEB_20230614.zip?sv=2017-04-17&se=2023-06-14T14%3A45%3A19Z&sr=b&sp=r&sig=ZjPp6Gh%2BYKMVdYrb2Xdb8IOxJcZKUrkfhY/cWWVeao4%3D

But the links generated yesterday (or) last week is working fine.
Tag Id :- program_602512d8e6aefa27d9629bc3:01269878797503692842
Link :- https://devbbprivate.blob.core.windows.net/reports/ml_reports//program-user-exhaust/D89883B9C00094E94742EE2A949857EC_20230613.zip?sv=2017-04-17&se=2023-06-14T14%3A44%3A40Z&sr=b&sp=r&sig=JHGamHhNTJxad58Ao8B1yWeS6s6CkuXO53fPMfPp2Hw%3D

[Shakthieshwari]

@AmiableAnil Thanks for the call today, As we discussed please find below the summary of the issues :-

1. Update Preference API is not working, Getting 403 Forbidden. As you requested, I have shared the curl command in the slack.
   Answer :- L3 & L4 update preference API is working, But for L1 (default) & L2 is not working. @AmiableAnil will check with JP and get back to us

2. PR Raised for the Data Product is still not merged because of Circle CI Failing, In my local Build is success. You told you will check with reshmi and get back to me, But still their is no any update on it
   https://github.com/Sunbird-Lern/data-products/pull/40/files
   Answer :- Reshmi and Anil is checking on this ...

3. Spark machine and Rancher access we got, but not sure how to use it and where to find out the logs, Request you to please help us
   Answer :- This is sorted, Anil & Kumar Helped

4. Tried to submit the job request with default(org) as the value for the key x-channel-id. But API is failing with 500 Internal Server Error. Shared you the curl in slack
   Answer :- Get the Valid Org Id from the Dev Lern Env and pass that org Id in the Submit Job Request API.
   Note :- Org Id should not be configured in the tenant preference API, Only then default preference value will be taken for this org Id when running the Data Product
   Will test this and get back..

5. Data Product Download Url Link is not working, giving me BlobNotFound Error. Attaching the screenshot below
   Answer :- Anil will check with devops team and get back to us on this ....

Cc- @aks30 @vijiurs

[rhwarrier]

@AmiableAnil @kumarks1122 can you pls check ?

[AmiableAnil]

@Shakthieshwari @aks30 Please plan to deploy dependent program related service in Lern dev env for the ML changes in data-product.

cc: @rhwarrier @reshmi-nair

[Shakthieshwari]

@rakeshSgr @aks30 FYI, Please take this forward