This repository contains sample code to make it easier to extend the context length of the Falcon 40B model. Please read the blog post for further instructions.
This code may increase the context length of the Falcon-40b-sft-top-1-560. Before using this code, or the Falcon model, you should perform your own independent assessment, and take measures to ensure that your use would comply with your own specific quality control practices and standards, and that your use would comply with the local rules, laws, regulations, licenses and terms that apply to you, and your content. The author of this code does not make any representations, warranties or guarantees that your use of this code or the model will result in a particular outcome or result.
See the medium blog post:
https://medium.com/@chenwuperth/extend-the-context-length-of-falcon40b-to-10k-85d81d32146f
This code is released under the Apache 2.0 software license
This code is a modification of the layers.py released by HuggingFace under an Apache 2.0 license in the following location:
