Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature planning: first class Python support #335

Closed
christopheranderson opened this issue May 4, 2016 · 50 comments
Closed

Feature planning: first class Python support #335

christopheranderson opened this issue May 4, 2016 · 50 comments

Comments

@christopheranderson
Copy link
Member

@christopheranderson christopheranderson commented May 4, 2016

This is a tracking item for First Class Python Support. We'll be tracking the requirements and issues for first class Python support via this item. Feel free to leave comments about features/design patterns.

The Functions team and the Microsoft Python team have engaged in a partnership to investigate first class Python support for the Azure Functions runtime.

Programming Model : Basics

Azure/azure-functions-python-worker#2

Programming Model : Data Types

Azure/azure-functions-python-worker#3

HTTP Triggers and Bindings

Azure/azure-functions-python-worker#4

Execution Context

Azure/azure-functions-python-worker#5

Logging

Azure/azure-functions-python-worker#11

Environment variables

Azure/azure-functions-python-worker#12

Version & Package management

Azure/azure-functions-python-worker#6

Testing/CI

TBD

Samples

Out of scope / Nice to have

1. Entry point decorator

Azure/azure-functions-python-worker#8

2. Access to bindings via context

TBD

Change log

5/5 : Chris - Add basic list of feature areas
1/10 : Asavari - Elaborating on the feature areas and programming model
1/11 : Asavari - Adding info about package management and environment variables
1/18 - Adding Samples
1/19 - Incorporating community feedback
1/23 - Incorporating community feedback (part 2)
1/24 - Adding improvements to context and logger
1/25 - Splitting into smaller issues and moving to the Python worker repo

@christopheranderson christopheranderson self-assigned this May 4, 2016
@christopheranderson christopheranderson added this to the backlog milestone May 4, 2016
@edevil
Copy link

@edevil edevil commented May 5, 2016

Support a requirements.txt file and runtime specification for specifying Python 2 or 3.

@brettcannon
Copy link
Member

@brettcannon brettcannon commented May 5, 2016

I would take it further than "2 or 3" and make it be by major.minor version.

@anthonyeden
Copy link

@anthonyeden anthonyeden commented May 16, 2016

The first step may be to publish some Python-specific documentation and examples.

For the benefit of anyone else trying to do HTTP Functions with Python right now, here's some basic example code I threw together: https://github.com/anthonyeden/Azure-Functions-Python-HTTP-Example/

@ryanga
Copy link

@ryanga ryanga commented Aug 17, 2016

Add python examples for the Azure Functions DocumentDB bindings: https://azure.microsoft.com/en-us/documentation/articles/functions-bindings-documentdb/

@paulbatum
Copy link
Member

@paulbatum paulbatum commented Nov 29, 2016

Support 64-bit Python.

@davidebbo
Copy link
Contributor

@davidebbo davidebbo commented Nov 29, 2016

Note that short term, it's possible to use a custom Python version using these steps: https://github.com/Azure/azure-webjobs-sdk-script/wiki/Using-a-custom-version-of-Python

@rawlukMike
Copy link

@rawlukMike rawlukMike commented Feb 2, 2017

EDITED : NOT TRUE

Solution above is not working anymore. I tried every step but in the end as long as you run inside home\site directory, you will always end up with Python 2.7.8.

Also why Python is so slow? It can takes up to 20s only at importing additional library. I tried virtual env and uploading full directory and running it from C# running cmd: "python.exe parameter".
These should be instant, and yet is so long.

@davidebbo
Copy link
Contributor

@davidebbo davidebbo commented Feb 2, 2017

I just retested that technique and it works fine. Did you make sure to expand the zip as you dropped it in D:\home\site\tools?

The better perf will come with the first class support that this thread is tracking. No ETA, but we will get there.

@rawlukMike
Copy link

@rawlukMike rawlukMike commented Feb 2, 2017

I am sorry for writing wrong. I tried Python 2.7.13 which have no embeddabled version.
Simply used zipped python with my site-packages.
Is there a way to use latest python 2.7.* with some libraries like pandas, xmltodict, numpy and so on... ?

@rawlukMike
Copy link

@rawlukMike rawlukMike commented Feb 2, 2017

Is there any special step for functions? I managed to install python through site extension gallery, but how can i bind it to functions? Basicly how to translate Configuring Your Site from guide above to functions :). Sorry maybe its a simple dummy question.

@brettcannon
Copy link
Member

@brettcannon brettcannon commented Feb 2, 2017

Unfortunately for Functions I don't know.

@davidebbo
Copy link
Contributor

@davidebbo davidebbo commented Feb 3, 2017

@rawlukMike the instructions I liked to shoudl work for any Python version. If you drop Python in the Tools folder it should get picked up because it's on the path.

@rawlukMike
Copy link

@rawlukMike rawlukMike commented Feb 3, 2017

Yep my bad. I used my python folder packed, but forgot msi installer puts python27.dll in system. Did not included it in zip. When I used unpacked but not installed folder from msi it works fine. Sorry for troubles.

@rawlukMike
Copy link

@rawlukMike rawlukMike commented Feb 3, 2017

Coping fully working Python 2.7.13 to the \home\site\tools is not working.
It worked for 3.x for some reason.

EDIT: After hour or more it started using Python from \home\site\tools

_@davidebbo You said that site\tools is in the PATH however Kudu show otherwise. It says '\site\deployments\tools' is in the path.
I unzip python in \home\site\deployments\tools (This one is IN PATH).
After that running in KUDU example: \home\python.exe -V started showing new one.
So it should work right?
Except its not.
I made simple Python function:
import sys
print(sys.version)
And it results with 2.7.8. With Python 2.7.13 installed in site\deployments\tools and being visible in Kudu as "primary".

So two questions here :).
1.Why even though KUDU is seeing Python from Path (actually there are two line in PATH pointing to python exe now : D:\home\site\deployments\tools, and D:\Python27), Functions still call old one?
2. Why coping Python 3.5 to D:\home\site\tools is working the first place since this one is not in PATH._

EDIT: I went to lunch and when I was back I saw functions using \home\site\tools Python 2.7.13
It just takes lots of time. My main confusion is multiple mentioning that \site\tools is in Path while kudu showed different. Sorry for troubling you guys. Keep it up with Functions they are fantastic :)

@davidebbo
Copy link
Contributor

@davidebbo davidebbo commented Feb 3, 2017

@rawlukMike site\tools is what's on the path for Functions execution. Kudu uses a different one which is for deployment tools, and is not on the path in the Functions runtime. Anyway, glad it's working for you now!

@rcarmo
Copy link
Member

@rcarmo rcarmo commented Feb 17, 2017

So I'm going down this particular rabbit hole, and keep bumping my head against the peculiarities of git deployment.

As nice as the portal UI is, I think there ought to be a hard requirement/user story that a Mac/Linux developer should only need to git push source code to the service to get a function going, and libraries be deployed either via Kudu (which is tedious and quirky, but passable) or automatically upon detection of the function config files and a requirements.txt.

I'm also not very keen on the current environment variable approach, since it limits the kinds of data functions have readily available.

There ought to be an import context from azure.functions (or similar) to abstract that away and provide a nicer way to deal with inputs and outputs, somewhat like the current JS API.

@rcarmo
Copy link
Member

@rcarmo rcarmo commented Mar 8, 2017

@rcarmo
Copy link
Member

@rcarmo rcarmo commented Mar 9, 2017

@quangv
Copy link

@quangv quangv commented Apr 14, 2017

sorry slightly off-topic, but what languages/os are currently a first-class citizen on Azure?

Just from a short Google search, I'm seeing:

.Net
Node.js
Java
Docker
Linux
Windows (assuming)
Open-Source

is anything else a first-class citizen? Or where can I find out that information.


@paulbatum Sorry, I moved my question to StackOverflow here.

@paulbatum
Copy link
Member

@paulbatum paulbatum commented Apr 14, 2017

@quangv this issue is not the appropriate forum for such a general question. Please keep this issue focus on Python. Thanks!

@oribarilan
Copy link

@oribarilan oribarilan commented May 15, 2017

Looking forward to Python support in VS Tools for Azure Functions and of course for Python templates in the Azure portal. Thanks!

@Krande
Copy link

@Krande Krande commented May 20, 2017

Hey, could you provide some sample code for python using a blob storage trigger of a specific file and performing some operations on the content (say a simple textfile/xml/json) and return a new/updated version of the file back to the storageblob?

EDIT

Nevermind, I figured it out using the following code:
PyFuncnBlob.zip

Anyways, keep up the good work with Functions and Python!

@oribarilan
Copy link

@oribarilan oribarilan commented May 20, 2017

@Krande - Did this sample code works for you using VS Tools For Azure Functions? Or do you use this inside the portal?

@Krande
Copy link

@Krande Krande commented May 20, 2017

I used the portal for testing the sample code.

@paulbatum
Copy link
Member

@paulbatum paulbatum commented Jan 12, 2018

@Krande It depends on what you mean by supported exactly. The experimental feature of creating python based functions still exists in functions v1 and it uses python 2.7.8. However as I mentioned above, there are significant performance issues with it and we don't plan to bring that implementation into general availability with full support, etc. This issue tracks implementing python support in functions v2 with an implementation that would have good performance characteristics and can be fully supported. The current plan would be for this implementation to target python 3.6, but I think there still needs to be more discussion and planning around this topic (can we support multiple python versions, etc).

@christopheranderson
Copy link
Member Author

@christopheranderson christopheranderson commented Jan 12, 2018

We are planning on making heavy use of the async library, which requires Python 3.4 or higher. That informs our current desire to avoid 2.7, but would love feedback on specific cases where 2.7 support would be required and 3.4+ wouldn't work.

@berndverst
Copy link

@berndverst berndverst commented Jan 13, 2018

Pipenv and Python 3.6+ seem like the way to go IMO. For anything else customers will still be able to bring their own containers.

I imagine a lot of people will still provide a requirements.txt instead of Pipfile or Pipfile.lock. In that case I recommend installing packages via pipenv install -r requirements.txt. In the case where either Pipfile and/or Pipfile.lock are provided, pipenv install should do the trick.

@Krande
Copy link

@Krande Krande commented Jan 15, 2018

Okay, I believe most of my python 2.7 code should be relatively easy to port to python 3.6. My experience with python 2.7 on azure functions is that once I add packages such as numpy and scipy I see a performance decrease running functions. Would that by any chance be related to the performance issue you mention (paulblatum) in v1? If so and upgrading to v2.0 and using python 3.6 fixes that then I welcome the change!

@asavaritayal asavaritayal removed this from the Unknown milestone Jan 15, 2018
@asavaritayal asavaritayal added this to the Backlog milestone Jan 15, 2018
@brettcannon
Copy link
Member

@brettcannon brettcannon commented Jan 15, 2018

@Krande I'm not on the Functions team, but based on how they have explained the v2 infrastructure to me I believe it will definitely be improved in v2 as functions will be imported once and so any start-up/import overhead will be cut out.

@pragnagopa
Copy link
Contributor

@pragnagopa pragnagopa commented Jan 16, 2018

@asavaritayal - over all looks good.

  • Add sample functions.json that uses type property - "Alternately, you can define the data type using the type property in the function.json configuration"
  • Update table under Context section to use context instead of executionContext
  • Need more details on what different log levels mean. For example, see NodeJS Logging and how to they can surface in AppInsight logs. Also, host.json has a tracing setting we need to use that instead of defining "logging"
@asavaritayal
Copy link

@asavaritayal asavaritayal commented Jan 19, 2018

Issue update: Incorporating community feedback and adding some samples.

@brettcannon
Copy link
Member

@brettcannon brettcannon commented Jan 22, 2018

One thing I realized is my suggestion to put the main function in a __main__ module might be too "cute" as it would prevent people from conveniently e.g. starting a test instance by pointing Python at the directory that contains your function code. Not sure what others think or what module name I would recommend instead (I know Asavari has suggested index.py to match the Node side), but I'm starting to think my initial suggestion for this was bad.

@1st1
Copy link

@1st1 1st1 commented Jan 22, 2018

A few thoughts/questions:

  1. We can support both async- and sync- functions: if the main() function is synchronous (no async qualifier) we automatically run it in an asyncio thread-pool:

    # Would be run with asyncio directly
    async def main():
        await some_nonblocking_socket_io_op()
    
    # Would be run in an asyncio thread-pool
    def main():
        some_blocking_socket_io()

    This approach gives the user a choice of what libraries they want to use. This way async/await becomes an opt-in.

  2. I agree with @brettcannon that using __main__.py module can be suboptimal. __main__.py has a very special meaning in Python: if an 'abc' package has a __main__.py file, it gets run on python -m abc. So a Python package may have a few __main__.py files in it, that are not intended to be used by Azure Functions. We probably want to pick another name to avoid potential problems here. By the way, is it required to have a function.json file per runnable Python function?

  3. context argument: is it optional? Do we want to auto-detect it? If it's the latter, we can make it a keyword-only required argument:

    # The 'context' argument will be injected automatically
    async def main(my_input: str, *, context: Context):
        pass

    We can do the same for context.logger—decouple it from the Context and pass it as a separate keyword-only argument:

    # 'context' and 'logger' arguments will be injected automatically
    async def main(my_input: str, *, context: Context, logger: Logger):
        pass
  4. Will we have a dedicated Azure Functions user module? That way we could provide functions to capture the current Context object instead of injecting it as an argument:

    import azure_functions
    
    async def main():
        context = azure_functions.get_context()

    The package could also contain helpers and fixtures for writing unittests.

  5. Having an explicit entry point decorator is a neat idea. Maybe this should be the one and only way of marking a Python function as runnable? I.e. it's perfectly fine to have a Python file with many helper functions none of which should be directly runnable:

    import azure_functions
    
    # Private helper function
    def helper():
        return 42
    
    # Directly runnable function
    @azure_functions.export  # or @azure_functions.runnable
    def main():
        return helper()
@asavaritayal
Copy link

@asavaritayal asavaritayal commented Jan 23, 2018

Thanks Brett and Yury for taking the time to think through this. Some notes below -

  1. Agree that we should support both async and sync, to allow flexibility of library and enable straightforward demo scenarios. That said, our docs should emphasize why it's important to use async to get more out of the language worker. I'll add this to the docket for later.

  2. @1st1, @brettcannon - A few options are: main.py or __run__.py . Let me know if one of these seems to resonate more, or you have other suggestions. To answer the second part : yes, we do require a function.json file per Function to define the corresponding bindings and their attributes. In fact, the portal UI lets you edit this file in the 'Integrate' tab beneath each function seperately.

  3. Since logger is going to be the only way to write trace outputs to the console, it makes sense to decouple it from context and provide as a keyword-only required argument.

  4. Open to the idea of having a dedicated module for Azure Functions. We've done this for a few other languages in the past and can discuss the pros/cons.

  5. For consistency across the board, we'd like to keep the entryPoint attribute of function.json. Given the time, I'd love for us to have a decorator and promote it as the primary way to declare an entry point in the script.

@1st1
Copy link

@1st1 1st1 commented Jan 23, 2018

@1st1, @brettcannon - A few options are: main.py or run.py . Let me know if one of these seems to resonate more, or you have other suggestions. To answer the second part : yes, we do require a function.json file per Function to define the corresponding bindings and their attributes. In fact, the portal UI lets you edit this file in the 'Integrate' tab beneath each function seperately.

If we require a function.json file, then maybe we don't need any special filenames for Python scripts? We can require the user to specify the name of the Python file and the name of the function in it in their function.json file.

@rcarmo
Copy link
Member

@rcarmo rcarmo commented Jan 23, 2018

@brettcannon
Copy link
Member

@brettcannon brettcannon commented Jan 24, 2018

The idea that @1st1 had about just putting it into function.json works for me. That would let people specify the dotted name to the callable object and thus provide anything that they want to call, including an instance of an object for a specific method, e.g.:

# In mypkg.mod
class MyFunc:
   def main(self, *args, **kwargs):
      ...

main = MyFunc.main

And then in function.json:

...
'function': 'mypkg.mod.main`
...

That way users have more flexibility and the only real requirement is that the object be callable.

@asavaritayal
Copy link

@asavaritayal asavaritayal commented Jan 25, 2018

We're moving this issue to its own repo - https://github.com/Azure/azure-functions-python-worker. For comments / feedback, please find the relevant issue in the new repo.

@sergei3000
Copy link

@sergei3000 sergei3000 commented Apr 13, 2018

The links under "Samples" in the opening post are not working

@BhaskarDasari
Copy link

@BhaskarDasari BhaskarDasari commented Jul 18, 2018

hi can you support requests module inside functions?

Hi azure function is not recognising the requests module . pasted the error and code please suggest the fix

exception:
File "D:\home\site\wwwroot\test\run.py", line 1, in
import requests
ImportError: No module named requests
.
python code:
import requests
import timeit
import datetime

def run_gomupload():
try:

 start_time = timeit.default_timer()
 cdate =datetime.datetime.now()
 print('Cell Start Time: '+ str(cdate))
    
 # code you want to evaluate
 elapsed = timeit.default_timer() - start_time

 cdate=datetime.datetime.now()

 print('\nCell End Time: '+ str(cdate))

 print('Cell Run Duration: '+ str(elapsed))
 # Clean up resources. This includes the container and the temp files
 #os.remove(full_path_to_file) 

except Exception as e:
print(e)

@paulbatum
Copy link
Member

@paulbatum paulbatum commented Jul 25, 2018

@BhaskarDasari - please post your question to stack overflow, msdn forum, or as a separate issue in https://github.com/Azure/azure-functions-python-worker - thanks!

@asavaritayal
Copy link

@asavaritayal asavaritayal commented Nov 6, 2018

Closing thread as the discussion has moved to the repo here - https://github.com/Azure/azure-functions-python-worker/issues

@msftbot msftbot bot locked as resolved and limited conversation to collaborators Jan 2, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet