[BUG] Homomorphic encryption with simulator

[Sahilnalawade]

Describe the bug
I was executing the Homomorphic encryption workflow on the Transformer Model. I was using the shareable_geenerator as HEModelShareableGenerator and aggregator as HEInTimeAccumulateWeightedAggregator in the server_config.json file. And teh task_result_filters used were HEModelEncryptor and HEModelDecryptor.

This workflow was executed on a Bert based model on NLP dataset, and I did encounted error message.
This entire run was simulated using the command, "nvflare simulator -w /tmp/nvflare/ -n 2 -t 2 nlp_task_he", where 'nlp_task_he' was the name of the package.

I have attached the error message (screen capture), also I have uploaded the error logs for the server and both sites.

To Reproduce
Steps to reproduce the behavior:

1. nvflare simulator -w /tmp/nvflare/ -n 2 -t 2 nlp_task_he
2. Error Message -

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots

Logs
server_log.txt
site1_log.txt
site2_log.txt

Versions - Dependencies

• Python Version [e.g. 3.9]
• NVFlare Version [e.g. 2.3.1]

[chesterxgchen]

@Sahilnalawade thanks for reporting this issue. We will take a look. @holgerroth is this related to : #1959 ?

[holgerroth]

Yes, this is a related issue as HE is not supported in simulator. @Sahilnalawade, please follow the example here to run an HE experiment with local provisioning: https://github.com/NVIDIA/NVFlare/tree/main/examples/advanced/cifar10/cifar10-real-world#31-secure-fl-workspace

[Sahilnalawade]

Hi Holgerroth,
Thank you for the feedback,
I tried running the above workflow locally.
I have few questions,

1. I am not sure what is the exact steps to follow to run the automated experiments.
2. Configuration, I am running on a non-GPU machine.
3. How do I monitor the runs, are there logs generated to check the runs for the experiment.
4. Where would the directory be located, '[DOWNLOAD_DIR]/[JOB_ID]/workspace/cross_site_val/cross_val_results.json'.
5. Normally, I would run the 'nvflare simulator' to run the experiment, is there a similar way to run the HE experiment.
6. Lastly, how do you know if the run is successfull, nor mally in the simulator mode, you can monitor the logs. How to do that with the above experiment for the cifar-10-real-world

[Sahilnalawade]

Just to add to the above point, I have updated the nvflare package to version 2.3.1, Do let me know, if you want me to test with a different version.

[holgerroth]

HE is not directly supported in the simulator at this point (see #1959). You'll need to use the admin console to monitor jobs and download results. The results will be available after download in your admin folder inside the secure workspace that you created.