PR for #67: Julia extension

[jc-cisneros]

This pull request closes gslab-econ/gslab_make#54 and #67. This template branch is currently pulling the 54-create-run_julia branch of the gslab_make submodule, so gslab_make PR for 54 should be closed first.

The proposed steps to test the new Julia wrapper are the following:

• Clone template and switch to 67-julia-extension branch
• Create the template_julia conda environment (i.e., run conda env create -f setup/conda_env.yaml from root)
• Load the submodule by running:

git submodule init
git submodule update

• Navigate to lib/gslab_make and run git switch 54-create-run_julia (we are testing the 54-create-run_julia branch of gslab_make). After that, run git branch. The current branch of the submodule should be 54-create-run_julia.
• Go back to root and activate the template_julia environment. Run check_setup.py from the setup module.
• Run julia julia_conda_env.jl from root to install the Julia packages in the conda environment (these are managed by Julia's built-in Pkg). Following the same logic of the conda_env.yaml file, the julia_conda_env.jl file installs the latest versions of the packages and has the versions used in the last working run commented out.
• Run run_all.py from root. A Julia script will be called on the analysis module. The file julia_scatter.png should be created and saved in analysis/output

Once the tests are successful, we can proceed to merge the 54-create-run_julia branch of gslab_make to master (through gslab_make PR for 54), point the submodule to master and run the entire repo again.

cc. @gentzkow @snairdesai @Houdanait @meyer-carl

[snairdesai]

@jc-cisneros I think we will need to update the submodule to point to the development branch commit you reference; it is currently still pointing to the latest commit in master. I'll re-review following this change.

[jc-cisneros]

@snairdesai I updated the instructions to fetch the correct submodule. The additional error you were getting regarding the packages (e.g. run_julia() throwing the error that it did not find any of the install packages) has been fixed on my end. In particular we want to run julia --project=.. <filepath> instead of julia <filepath>. By doing that, we use the Julia environment that was instantiated in root.

[snairdesai]

Thanks @jc-cisneros! I'll proceed with the review then.

[snairdesai]

@jc-cisneros Seems like this is closer to running. I'm getting the following errors, the first of which originates when I attempt to instantiate our objects in the interactive Julia terminal, and the second (which I believe follows directly from the first) is called when I execute run_all.py. Any feedback on resolving this (I've already rebuilt the template_julia environment just in case)?

(1)

---

(2)

[jc-cisneros]

Thanks @snairdesai ! I rebuilt the Manifest.toml and Project.toml files. I uninstalled my local Julia and cleared my .julia cache. The run was successful in my end. Let me know if it works for you @snairdesai.

[snairdesai]

@jc-cisneros still no luck - here's the new error messages from the PyCall build:

[jc-cisneros]

@snairdesai I changed the process to build the Julia environment using a script instead of the .toml files. The Manifest.toml and Project.toml files will now be created in the /conda/base/envs/template_julia/share/julia/environments/template_juliadirectory (i.e., the Julia installed in the conda environment is managing a template_julia Julia environment with Pkg). Let me know if everything works on your end.

[snairdesai]

@jc-cisneros No luck - errors seem a bit more severe. I've attached some screenshots below:

---

---

---

---

---

[jc-cisneros]

@snairdesai I have not encountered this error on my end, but my current best guess is that it might be related to the Python distribution the Julia environment is pointing to. The error message states that it is indeed using the Python distribution in the Conda package, but it might be using a Python distribution you have on your miniconda/base/bin instead of the one in your template julia environment. Let me test a possible solution on my end, and I will let you know when you can give it another try. Thank you!

[jc-cisneros]

@snairdesai building the Conda environment + Julia environment setup has been tricky (trickier than writing the wrapper). I have completed a full run on a Docker Debian container (8b1c26b) and on my local Mac OS afterwards (4863c67). The Conda.jl package should be capable of dealing with the Python path issue. Let me know if it works on your end.

[snairdesai]

@jc-cisneros Flagging I have the same errors as in the comment above.

---

[jc-cisneros]

Thanks @snairdesai! Let me explain the latest changes:

• All of the errors above are caused by the PyCall.jl package. By default, this package uses the Python version in the Conda directory. This might lead to issues when there are several Python versions living in the Conda directory (i.e. there are several environments). We can fix this in an interactive Julia session by changing the path of the Python Environment variable. Nevertheless, to make this change global we might have to touch a user configuration file after creating the Conda environment (see https://docs.julialang.org/en/v1/manual/environment-variables/). I think being able to call Python functions within Julia is not a feature we really need (we could simply run Python scripts within the same project), so this effort is not worth pursuing and it is out of the scope of this PR.
• The new test script does not use the matplotlib backend for the plot. @snairdesai let me know if the version conflicts are gone and if the run_julia wrapper works.

[snairdesai]

@jc-cisneros We're very close here - the plots are compiling now. Small issue, I had another terminal window open on my local computer when the Julia script ran, which I had to manually exit out of for the code to finish compiling. Ideally we would want this to automatically exit for the user. A screenshot of the outputs are below:

---

[snairdesai]

@jc-cisneros Post your latest commit a45a0a2 - this compiles successfully! I'll proceed to a code review now.

[snairdesai]

@jc-cisneros For a future PR (perhaps not this one), we probably want to:

1. specify versions of these packages explicitly, to maintain the transition to this format across R and Python
2. in our check_setup script, notify the user if they have not properly installed the correct Julia packages.

Just tagging this here.

[snairdesai]

@jc-cisneros

Interesting way to kill the terminal plot environment, agree it's the best we have for now.

[snairdesai]

@jc-cisneros

We'll want to track developments made by the conda-forge team such that when they have implemented the proper fixes, we migrate the calls to Julia and package installer directly to this conda_env.yaml file, rather than our separate julia_conda_env.jl script (much like we do for pip in R.

[snairdesai]

Small comment on the conda_env.yaml package versions, but otherwise this looks good! Hopefully the conda-forge developments coming down the pipeline will make this process easier.

[jc-cisneros]

@snairdesai thanks for the careful review! In the latest commit 8f0cec7, I added the versions of all packages used (as in master). I also commented out all the Julia-related lines, so that the default behavior in master is not installing Julia and producing the test plot. Throughout all these tests, the run_julia wrapper has been working as expected. I will stress in the final summary that until the julia-feedstock is further developed on conda-forge (see #67 (comment)), we will need this julia_conda_env.jl workaround (which as shown in this PR, can create conflicts if Python is called from Julia).

If everything looks good, I will proceed to merge the development branches of this PR and the PR on gslab_make to their respective master branches. The final step then would be to add summaries that close #67 and #54 on gslab_make.

[Houdanait]

Hey @jc-cisneros @snairdesai !
Thanks so much for all the brilliant work here !
Please ping me once you're done merging this to the template as we will need this extension in another repo !

[snairdesai]

Thanks @jc-cisneros! The revisions look good to me - I'm approving this PR, and simultaneously the one for #54 on gslab_make.

[jc-cisneros]

Thanks again for the review, @snairdesai! I will close the issues with a summary and merge this branch to master.