Error importing Shared Code into Python function

[wbreza]

Hack required to import Shared Code modules into Python functions

Investigative information

Repro steps

1. Create a folder SharedCode adjacent to your function folders
2. Added a python module into the SharedCode folder
3. Attempt to import your SharedCode modules into existing Python Azure Function
4. Run your Python Azure Function

Expected behavior

Module should be available by default because it is inside the same virtual environment without requiring any PYTHONPATH mods.

Actual behavior

Error:

Microsoft.Azure.WebJobs.Host.FunctionInvocationException: Exception while executing function: Functions.HttpTriggerPython ---> Microsoft.Azure.WebJobs.Script.Rpc.RpcException: Result: Failure
Exception: ModuleNotFoundError: No module named 'SharedCode'

Known workarounds

Add the following lines to the top of your __init__.py in your Azure Function. This will allow any modules defined in your FunctionApp to be available for import

import sys
import os
sys.path.append(os.path.abspath(""))

[elprans]

You don't need any hacks. Import adjacent modules using the relative import syntax:

from . import SharedCode

@asavaritayal Perhaps we should document this in a conspicuous fashion, since this is not the first time this issue has come up: #81

[wbreza]

@elprans Not working for my case.

from . import SharedCode results in error Exception: ImportError: cannot import name 'SharedCode'

I would like to import a submodule using the following or similar syntax
from SharedCode import MyTestClass where MyTestClass is a class defined in module1.py for example.

Folder structure looks like the following

FunctionApp
|_ HttpTrigger1
    |_ __init__.py
    |_ function.json
|_ HttpTrigger2
    |_ __init__.py
    |_ function.json
|_ SharedCode
    |_ __init__.py
    |_ module1.py
    |_ module2.py

[elprans]

@wbreza In your case your shared module code is one level up, so the correct syntax would be

from .. import SharedCode

[elprans]

To import submodules directly:

from ..SharedCode import module1

[tbarlow12]

@elprans I changed all of our imports to relative imports like you indicated above:

from ..Common import ResourceTagger
from azure.functions import QueueMessage
import ..Adapters

def main(msg: QueueMessage):
    ResourceTagger.process_queue_message(msg)

I get the following error:

Exception while executing function: Functions.StoreResources. System.Private.CoreLib: Result: Failure
Exception: SyntaxError: invalid syntax
Stack:   File "C:\Users\tabarlow\source\virtualenvs\resource-scanner\lib\site-packages\azure\functions_worker\dispatcher.py", line 202, in _handle__function_load_request
    func_request.metadata.script_file)
  File "C:\Users\tabarlow\source\virtualenvs\resource-scanner\lib\site-packages\azure\functions_worker\loader.py", line 62, in load_function
    mod = importlib.import_module(fullmodname)
  File "C:\Users\tabarlow\source\virtualenvs\resource-scanner\lib\importlib\__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 941, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 674, in exec_module
  File "<frozen importlib._bootstrap_external>", line 781, in get_code
  File "<frozen importlib._bootstrap_external>", line 741, in source_to_code
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed

This does not occur when using the hack described above:

import sys
import os
sys.path.append(os.path.abspath(""))

Just for reference, this is running on a clean virtualenv with the following dependencies:

gazure-cosmosdb-table
azure-functions==1.0.0a4
azure-functions-worker==1.0.0a4
grpcio==1.14.2
grpcio-tools==1.14.2
protobuf==3.6.1
six==1.11.0
azure-mgmt-nspkg
azure-mgmt-resource
azure-mgmt-subscription
azure-servicebus
azure-storage
azure-storage-common
elasticsearch
click==6.7
PyMySQL
python-dotenv
pytest
pytest-cov

[elprans]

@wbreza Change import ..Adapters to from .. import Adapters

[wbreza]

@elprans - Thanks - this does work but based on Python best practices that I have read relative imports are frowned upon. Some pros/cons and discussion @ https://realpython.com/absolute-vs-relative-python-imports/

I am not a python expert myself but in refactoring our project to support relative import it caused many downstream issue with how pytest and other modules are invoked.

Any idea how to make this process easier by allowing Absolute imports?

[elprans]

@wbreza If you have a lot of shared code, the best approach would be to put it into a standalone pip-installable package and add that as a dependency of your function app. This way you would be able to perform absolute imports.

[priyaananthasankar]

@asavaritayal is there a plan to record these best practice recommendations in the developer guide? Especially since these are idiomatic to Python.

[wbreza]

@elprans - Take your point, but keep in mind not all consumers of this will be able to push their IP to pypi unless they could host it privately. We may unfortunately actually just stick with our path workaround mentioned above.

[elprans]

@wbreza We can add support for building dependencies from local sources when publishing. Pushing to PyPI should not be necessary, all you'll need to do is put a path in requirements.txt:

public-package==1.0.0
...
file:///my/proprietary/code

or, if you have pre-built wheels:

public-package==1.0.0
...
file:///my/proprietary/code-2.0.whl

We chose against adding the function app folder to sys.path because the module namespace in Python is flat, so if you name your function numpy, you wouldn't be able to import the real library and this may cause massive confusion, especially if the masked module is not a direct dependency.

[asavaritayal]

@asavaritayal is there a plan to record these best practice recommendations in the developer guide? Especially since these are idiomatic to Python.

Yes, we should add this to the developer guide. @elprans can you update the wiki?

[elprans]

@asavaritayal Done.

[asavaritayal]

@elprans, @brettcannon ran into the same issue yesterday. Is there a way we can work around the relative import syntax for shared modules?

[elprans]

@asavaritayal, @brettcannon What exactly is wrong with relative imports?

[brettcannon]

@elprans it leads to an odd project layout. For instance, my project is now laid out as:

• pvscbot
  • __init__.py (needed, else pytest isn't happy)
  • requirements.txt
  • github/
    • __init__.py
    • ...
  • ghutils/
    • __init__.py
    • ...

I'm working entirely within the pvscbot directory. So I have pvscbot opened as the directory in VS Code, but I'm needing to run pytest as python -m pytest .. to execute the tests so it knows where the top of the package is. And then embedding my requirements.txt file is odd when within a package. It also means func init has to tell users to name that directory it creates something that can be imported (I initially named it pvsc-bot).

I also don't know how I want to check this into GitHub. I.e. do I check in pvscbot as the top directory in the GitHub repo even though there won't be anything else but that directory and a README, or do I check in the contents of pvscbot as the top of the repo and then tell people they will want to clone into a sub-directory?

[elprans]

OK, but the function app directory layout isn't really dictated by the worker, it's how host/CLI expect it to be. The issue at hand is how you address the Python modules that comprise your function app (which is currently to import them using the relative import syntax). Am I missing something?

[brettcannon]

Basically if the docs get updated to say "put everything in a single package" then that's fine but then I want to move requirements.txt up a level and I want func azure functionapp publish to work from the directory above as well. IOW if I can't have individual functions treated as individual packages and instead need to provide an unambiguous package that contains all code then I want the following layout to work:

• requirements.txt (and be able to run func azure functionapp publish from here)
• tests/
• pvscbot
  • __init__.py (needed, else pytest isn't happy)
  • github/
    • __init__.py
    • ... including Function-related stuff
  • ghutils/
    • __init__.py
    • ... shared code

https://github.com/Azure/azure-functions-python-worker/wiki/Developer-Guide#folder-structure suggests that currently isn't possible.

[elprans]

I want to move requirements.txt up a level and I want func azure functionapp publish to work from the directory above as well.

I think that's reasonable. But then again, it's not up to the worker how the code is laid out. The host dictates the path to the function directory, and that's it.

[elprans]

To clarify, we currently employ this importlib hackery to make even the relative imports work properly:

azure-functions-python-worker/azure/functions_worker/loader.py

Lines 23 to 29 in 5ce1d21

	def install():
	if _AZURE_NAMESPACE not in sys.modules:
	# Create and register the __azure__ namespace package.
	ns_spec = importlib.machinery.ModuleSpec(_AZURE_NAMESPACE, None)
	ns_spec.submodule_search_locations = _submodule_dirs
	ns_pkg = importlib.util.module_from_spec(ns_spec)
	sys.modules[_AZURE_NAMESPACE] = ns_pkg

Again, this is because we can't really rely on the layout or the setting of PYTHONPATH.

[brettcannon]

@elprans thanks for the details!

@asavaritayal not sure if these layout requirements can change, but that's my feedback about where you are currently asked to place files.

[maiqbal11]

The following issue has the approach that we'll be using for handling absolute imports with our current layout: #278. Docs on this coming shortly. Closing this as they have significant overlap.

[polarapfel]

To import submodules directly:
from ..SharedCode import module1

I still don't get it. I followed the instructions in the documentation page (matching the quoted comment here).

I get this in pylint and at runtime:

Attempted relative import beyond top-level package

Clearly, the documentation is wrong. What is the right way to share code between multiple functions in the same app? Can someone please fix the documentation?

[maiqbal11]

Hi @polarapfel, could you create a separate issue? We can follow up there with further details.

[ghost]

@maiqbal11 I logged the issue here MicrosoftDocs/azure-docs#30588

[bigdatamoore]

I was having the same issue as the original poster, and was able to get things working by adding the sys.path.append workaround. However we have found that fails occasionally when deployed with an error of unable to import module.

So I tried the method suggested by @elprans:

from ..SharedCode import module1

That works but now my tests are failing. So, my question is now, what is the recommended folder structure for unittests? The folder structure in the python function developers guide doesn't talk about a test folder.
https://docs.microsoft.com/en-us/azure/azure-functions/functions-reference

[anindya5]

Hi,

My issue is if I want to import a function say function1 in module1 of SharedCode, but for me the SharedCode is inside the API folder, like below

FunctionApp
|_ HttpTrigger1
     |_ __init__.py
     |_ function.json
     |_ SharedCode
        |_ module1.py
        |_ module2.py
|_ HttpTrigger2
     |_ __init__.py
     |_ function.json

Usually I'd have done
from ShareCode.module1 import function1

That does not work.

from .ShareCode.module1 import function1 also gives module not found error, shown below

Executed 'Functions.HtmBatchAzureFunc' (Failed, Id=d70ee3dc-dca2-4d89-b39c-5c902ed64b11)
[5/22/19 8:54:05 PM] System.Private.CoreLib: Exception while executing function: Functions.HtmBatchAzureFunc. System.Private.CoreLib: Result: Failure
[5/22/19 8:54:05 PM] Exception: ModuleNotFoundError: No module named 'htm_shared_src'
[5/22/19 8:54:05 PM] Stack: File "/usr/local/Cellar/azure-functions-core-tools/2.7.1158/workers/python/deps/azure/functions_worker/dispatcher.py", line 229, in _handle__function_load_request

Can anyone help

[brettcannon]

@anindya5 which module in your example file hierarchy are you trying to do the import from? Also note you don't have an __init__.py in your SharedCode/ folder. Is that on purpose?

[jaymegordo]

I had the same issue with needing absolute imports.

sys.path.append(os.path.abspath("")) > did not work
sys.path.append('/home/site/wwwroot') > did work

[bhbitter]

Same thing is happening to me now. I had a function that was deployed and working fine. Now the same code stop working with this error. I didn't change or deploy anything. I had to add the path append like @jaymegordo.

sys.path.append('/home/site/wwwroot')

[anindya5]

Codes are not finding or navigating to proper files suddenly everywhere

On Mon, Aug 12, 2019 at 7:22 AM Brad Bitterman ***@***.***> wrote: Same thing is happening to me now. I had a function that was deployed and working fine. Now the same code stop working with this error. I didn't change or deploy anything. I had to add the path append like @jaymegordo <https://github.com/jaymegordo>. sys.path.append('/home/site/wwwroot') — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#219?email_source=notifications&email_token=ACQLYQHHQMHA7I7LRP2JDM3QEFIXBA5CNFSM4FZXKBMKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD4CLLMI#issuecomment-520402353>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACQLYQBQPUGUFIK4K3VYR2LQEFIXBANCNFSM4FZXKBMA> .

-- Anindya Sankar Dey

[maiqbal11]

@jaymegordo, @bhbitter, @anindya5, it would be best if you are able to open separate issues with details/repro of the exact issue you are facing. There are different flavors of import issues that can happen and having separate issues with details will help us determine if there is some common cause and help keep our responses as specific as possible. Thanks!

[KikoRabacal]

facing same issue... all imports working with SharedCode.MODULE 2 weeks ago, now it's not working with error "Module not Found"

[dxkaufman]

@elprans I am having similar problems, described in detail in a new issue: #577. My function was working in my local environment on Oct 31, but is not working since upgrading VSC and Azure functions core tools to the latest versions in the last few days. I have tried changing from SharedCode import FunctionApp to from .SharedCode import FunctionApp and some other permutations, but anything that starts with from . gets me an error attempting a relative import beyond the top-level package. I've also tried putting all the code directories in an __app__ directory, but that hasn't helped because then it doesn't see the functions. I am completely stuck in the water, though I'm still sifting through google, stackoverflow & github searches hoping I'll find something that does it. Any help you can provide, I'd be enormously grateful! Thank you for your time!

[oleksandr-yatsuk]

A better workaround is to add to function's __init__.py this code:

import sys
from os import path

sys.path.append(path.dirname(path.dirname(__file__)))

With doing that absolute imports can be used

[jaymegordo]

@oleksandr-yatsuk THANK YOU for that

My imports randomly stopped working a day ago and using that syntax instead works!

now:
sys.path.append(os.path.abspath("")) > doesn't work
sys.path.append('/home/site/wwwroot') > doesn't work
sys.path.append(path.dirname(path.dirname(file))) > DOES work

[dxkaufman]

@oleksandr-yatsuk Good suggestion. I had to create that for each of my functions, but I like having it there, rather than in the import section in each code file.

[L480]

Just put your modules in the following directory and import them as usually:

FunctionApp
|_ host.json
|_ proxies.json
|_ .python_packages
   |_ lib
      |_ python3.6
         |_ site-packages
            |_ azure
            |_ urllib3
            |_ cryptography
|_ HttpTrigger1
   |_ __init__.py
   |_ function.json

[KantiCodes]

Hi guys,
TL DR I had the same issue, was getting on my nerve not being able to do relative/regular import fix -> uninstall pylint.

Description
I just switched from Pycharm to VS code for best serverless experience, downloaded the demo with HttpTrigger. Afterwards added a SharedCode folder to my project. (just like this folder structure from here)
Tried to import a file from SharedCode into init of my trigger but it failed.
Everything was under the same venv, could not figure what is wrong.
Then I found link
Uninstalled pylint -> imported the SharedCode file the same but this time it worked.
EDIT:
Adding this python.linting.pylintPath fixed the problem and now I can happily live together with pylint in my VS code :)

[SeaDude]

How are non-Python code such as .ttf font files referenced by a Function? I have some Python code that uses custom fonts to return nice barcodes with company info printed on them.

Outside of Azure Functions, I simply put the .ttf files in the project root directory and the code can reference them without issue. I tried doing with with my Azure Function, but the Function can't "see" them.

Since they are not .py files, I can't import them (right?), so how is this scenario handled?

Thank you

[SeaDude]

Disregard. Just had to include the folder path of the Function.
Was trying:
font = 'fontFile.ttf'
Needed:
font = 'folderHoldingInit/fontFile.ttf'