pytorch版本问题，运行整套流程torch版本需要特定的版本吗？还是说2.1.0以上就可以

[Joker-sad]

No description provided.

[Chillee]

It needs to be PyTorch nightly for now.

[merveermann]

It needs to be PyTorch nightly for now.

I use 2.3.0.dev20231214+cu121 version, but I got the error mentioned in #49. May it be related with the pytorch version? Which version do you use?

[learning-chip]

FYI, I am able to run this code repo using torch==2.1.2+cu121 (current stable release, not nightly), by just commenting out torch._inductor.config.fx_graph_cache = True in generate.py which is not available in this torch version. The rest of code can remain unchanged.

Reference run on RTX 4090

export MODEL_REPO=openlm-research/open_llama_7b
./scripts/prepare.sh $MODEL_REPO
python generate.py --compile --checkpoint_path checkpoints/$MODEL_REPO/model.pth --prompt "Hello, my name is"

Log:

with compile:

python generate.py --compile --checkpoint_path checkpoints/$MODEL_REPO/model.pth --prompt "Hello, my name is"
Loading model ...
/opt/conda/lib/python3.10/site-packages/torch/_utils.py:831: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
  return self.fget.__get__(instance, owner)()
Time to load model: 5.11 seconds
[2023-12-17 10:30:31,781] [0/0] torch._dynamo.output_graph: [WARNING] nn.Module state_dict and backward hooks are not yet supported by torch.compile, but were detected in your model and will be silently ignored. See https://pytorch.org/docs/master/compile/nn-module.html for more information and limitations.

Compilation time: 86.28 seconds
Hello, my name is Tom and I am a recovering alcoholic and addict. If you have watched The Mowgli's YouTube videos or listened to their sound on Spotify, you might be surprised to learn that the two guys singing are recovering addicts. In 2014, I got a chance to see them live in concert and was immediately struck by how their story and message was one of redemption. They have all been clean for a little over five years.
I grew up in a Christian background, so I have seen a lot of people fall away from the Lord and/or lose interest in their faith. However, I had never listened to a band that was so transparently positive with their testimony of God's grace that brought them to a better life. They are brothers, and they have been through the same struggles that I suspect many of us have. However, they have been able to find the hope and inspiration to move on, which is inspiring and encouraging.
So, I hope
Time for inference 1: 3.14 sec total, 63.63 tokens/sec
Bandwidth achieved: 857.55 GB/s
Hello, my name is Hannah I am eighteen and I am currently a university student.
I am a social butterfly and love meeting new people. I am an outgoing and bubbly girl, everyone always comments how much they enjoy my company (which I am so grateful for).
I am a great listener as I love to just sit back and listen to people talk.
I am open minded, I am also ambitious and have a strong work ethic. I love to try new things in my life and always keeping my mind open to new ideas and ways of thinking.
I am great at keeping in contact with people and I am very good at being on time. I love to make new friends as well as making new memories with people. I love to have fun with people as I love to laugh as much as I can.monyjewels Narcisse,  samaire, monsieur, nikki, vicky, hélène, patrick, la
Time for inference 2: 3.14 sec total, 63.66 tokens/sec
Bandwidth achieved: 857.89 GB/s
Hello, my name is Jonathan. I am an English teacher and an American expat living in Sao Paulo, Brazil. I am also the host of the Brazilian Portuguese podcast, Portuguese for Brazilians. In this podcast, I focus on Brazilian Portuguese, the second most spoken language in the Americas. If you are interested in Brazil, and interested in learning Portuguese, this is your podcast!
Listeners of Portuguese for Brazilians, please visit www.portugueseforbrazilians.com. </p>
                                        <p>This is a good place to start: <a href='https://www.youtube.com/watch?v=7MtqLN_19Yo'>How to Learn Portuguese (Brazilian) in 3 Simple Steps</a>.</p>
                                        <p>Not sure if you should learn Brazilian Portuguese or Portuguese from Portugal? <a href='https://hub.docker.com/
Time for inference 3: 3.14 sec total, 63.61 tokens/sec
Bandwidth achieved: 857.29 GB/s
Hello, my name is Martin Rotherham. I am an associate in the Commercial Litigation team at Berry Smith.
I have been involved in a wide variety of commercial litigation including debt recovery, contractual disputes, shareholder disputes, banking and professional negligence claims, as well as breach of confidence.
I also have a wide general common law litigation practice, having acted in cases in the High Court and County Court in England and Wales.
I have experience in dealing with banks and lending institutions, having acted for and against banks, both as claimant and defendant in litigation involving mis-selling of complex financial products, including mortgage-backed securities, residential and commercial investment trusts, and loan notes. I acted for the bank in a case in which the trustees of a mortgage-backed securitisation fund were successfully sued for negligence, breach of trust and breach of contract for losses arising from a default in a credit default swap.
In the banking field, I have also worked on claims relating to alleged mis-selling and negligence in
Time for inference 4: 3.14 sec total, 63.64 tokens/sec
Bandwidth achieved: 857.66 GB/s
Hello, my name is Chris and I am a bit of a car nut. Over the years I have owned a few cars and from time to time shared them with friends.
For the last three years I have owned a 1976 Ford Capri 2.8i. This car is a good example of how you can buy a car for a good price and make it very nice. However, it is getting a bit tired and I have decided to sell it.
In the meantime I own a 2013 Jeep Wrangler Rubicon Unlimited, which I have had since new.
I would like to sell the Capri to a person that is also a car nut. If you are interested drop me a message and we can have a chat. architec-1975.com (Must type the L's in blue), this is a very nice country-style home with 4 bedrooms, 2.5 baths, screened porch plus
Time for inference 5: 3.15 sec total, 63.57 tokens/sec
Bandwidth achieved: 856.78 GB/s
==========
Average tokens/sec: 63.62
Memory used: 14.14 GB

without compile:

python generate.py --checkpoint_path checkpoints/$MODEL_REPO/model.pth --prompt "Hello, my name is"
Loading model ...
/opt/conda/lib/python3.10/site-packages/torch/_utils.py:831: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
  return self.fget.__get__(instance, owner)()
Time to load model: 4.80 seconds
Hello, my name is Iris.
My name is Iris and I am the New Entry Project Officer at Wysing Arts Centre.
I am originally from Barcelona, and I have a background in English Literature and Creative Writing. I have worked as a Digital Marketing Assistant at the British Council in Spain, and as a Research Assistant at the International Studies Program at Universidad Autónoma de Barcelona.
During my time at university I focused on writing, and had the opportunity to set-edit a literary journal for two years.
What brought me to Wysing, and Cambridge in particular, is the programme for the New Entry Project, a three-year programme to foster the development of new artistic practices.
Over the past six months, I have been part of a team providing support for artists and musicians on the programme. This has involved coordinating the use of Wysing’s studios and facilities for the new entrants. contemptible  /*!*!*!*!*!*
Time for inference 1: 6.23 sec total, 32.12 tokens/sec
Bandwidth achieved: 432.90 GB/s
Hello, my name is Tomás. I am 24 years old and I am a student of Medicine at the Autonomous University of Madrid.
My interest in the world of cryptocurrencies began last May during a master's course on topics related to digital currencies in the University of Alcalá de Henares.
Since then I have investigated the development of bitcoin, so I came to discover Ethereum and I was amazed to see the possibilities of the platform.
I have seen that Ethereum has a real potential, so I decided to start investing and then sharing my knowledge with others.
My goal is to make accessible to everyone this new platform and its possibilities.
I hope you enjoy my videos and that you continue supporting me! COLD BANK CASH!!! Synthetic Gold is the best gold to invest in.
This entry was posted on November 22, 2017 at 3:07 pm and is filed under Uncategorized. You
Time for inference 2: 5.53 sec total, 36.18 tokens/sec
Bandwidth achieved: 487.54 GB/s
Hello, my name is Hannah and I'm 17 years old. I live in Columbus, GA with my mom and two stepsisters.
I am a junior in high school and am on the student council.
My hobbies are photography, graphic design and art.DT is my favorite shop.
I have been sewing for over 10 years, most of them self-taught.
I recently started selling items at the DT Marketplace so I was very excited to be chosen as one of the DT’s blog features! I was also super surprised when I got the notification as I wasn’t sure if I was being featured or not!
I’m not sure I have a favourite Dollybird…I love all of them and have met most of them either at DT events or on the DT blogs…I love Tracey’s blog too!
I love all aspects of the DT and would like to mention all of them for their hard work and passion…especially
Time for inference 3: 5.70 sec total, 35.10 tokens/sec
Bandwidth achieved: 473.01 GB/s
Hello, my name is Jonathan and I am a poet, a philosopher, a writer, and a thinker.
So to anyone that has been reading my blog for even a little bit, you know that I am a very analytical person. I consider a lot of things, people, and things to be important to me. I don't like to do things halfway, I want to make an impact on the world around me.
This blog is all about my thoughts, opinions, and philosophical reflections. I give up a lot of my personal life and conversations for everyone to read and hear, but I feel that this is what it's all about: I want to share my ideas with you, the reader. I want to tell you about what I think, and how I think, and what I believe and don't believe. I want to help, if I can, to make the world around me a better place.
I want to take what I'm doing here and take it a
Time for inference 4: 5.70 sec total, 35.08 tokens/sec
Bandwidth achieved: 472.77 GB/s
Hello, my name is Martin Steffens.
I am co-founder and co-CEO of WeGift.
Let me start by quickly introducing our company.
WeGift is a money transfer and payment platform that connects people and businesses.
Our company is backed by a venture capital fund and we have about 100 employees in Brno and Prague.
With that in mind, I will be talking about the design of our logo.
This is the first time that I am giving a talk on the design of a logo,
but it is an interesting topic, nonetheless.
You can find more information about our logo at our website.
So, let's dive in.
The first part of my talk will be a rough explanation of our logo.
The second part will be a detailed explanation of the details of the logo.
So, let's start with the rough explanation.
I'll try to keep it short.
Our logo is a
Time for inference 5: 5.63 sec total, 35.54 tokens/sec
Bandwidth achieved: 478.93 GB/s
==========
Average tokens/sec: 34.80
Memory used: 13.62 GB

Compile gives 63.62 / 34.80 = 1.8x speed-up