Get long lived download URLs in addition to SignedURLs

[tzvc]

Hi there,

I noticed that there is currently no way to get a long-lived download URL from an uploaded resource using the node admin package which causes problems when storing generated URLs in external DB as they expire after some time.

Would it be possible to generate long-lived URLs just like the getDownloadUrl() function from the client SDKs
(https://firebase.google.com/docs/reference/android/com/google/firebase/storage/StorageReference.html#getDownloadUrl() ).

Cheers!

[avin3sh]

From GCP Storage Docs,

Get the name of the bucket containing the public data.

Use the following URI to access an object in the bucket:

http://storage.googleapis.com/[BUCKET_NAME]/[OBJECT_NAME]

For example, the Google public bucket gcp-public-data-landsat contains the Landsat public dataset. You can link to the publicly shared object LC08/PRE/063/046/LC80630462016136LGN00/LC80630462016136LGN00_B11.TIF with the link:

http://storage.googleapis.com/gcp-public-data-landsat/LC08/PRE/063/046/LC80630462016136LGN00/LC80630462016136LGN00_B11.TIF

This link does not require authentication in order to use. It is suitable, for example, as a link in a web page, or for downloading with a command-line tool such as cURL.

[tzvc]

@avin3sh thanks for the reply, this is the kind of long lived URLs i'd be looking for, however, this only applies to public data.

[tdkehoe]

I write a book-length :-) summary of the problems associated with the three types of download URLs for Google Cloud Storage, on Stack Overflow. tl;dr the signed URLs expire in at most seven days (but the documentation doesn't say this, so your project crashes every week during the month or two you're trying to figure this out); making your file public isn't a good idea in every situation; and there's an undocumented property of the Storage object that enables setting your own token, which works but I hesitate to use undocumented features.

[AVaksman]

The maximum duration that the signed url should be valid for is mentioned here

[AVaksman]

Currently 7 day limit is only true for V4 signatures and API will throw a Max allowed expiration is seven days (604800 seconds). error.
V2 signatures still support longer duration.

[tzvc]

Currently 7 day limit is only true for V4 signatures and API will throw a Max allowed expiration is seven days (604800 seconds). error.
V2 signatures still support longer duration.

Do you have a link of the docs for that? I don't understand why it's not possible to have an infinite duration, it would be much needed for things like user profile pictures

[tdkehoe]

Currently 7 day limit is only true for V4 signatures and API will throw a Max allowed expiration is seven days (604800 seconds). error.
V2 signatures still support longer duration.

Version 4 of what software? Do you mean uuidv4?

[tdkehoe]

And what about v5? Does it have the 7 day limit?

[AVaksman]

Currently 7 day limit is only true for V4 signatures and API will throw a Max allowed expiration is seven days (604800 seconds). error.
V2 signatures still support longer duration.

Version 4 of what software? Do you mean uuidv4?

I meant 'v2' vs 'v4' signing with service account authentication (doc ref), can be specified inside config (API ref).

[AVaksman]

Currently 7 day limit is only true for V4 signatures and API will throw a Max allowed expiration is seven days (604800 seconds). error.
V2 signatures still support longer duration.

Do you have a link of the docs for that? I don't understand why it's not possible to have an infinite duration, it would be much needed for things like user profile pictures

It is documented here (X-Goog-Expires)

[giammin]

@AVaksman you closed the issue but did not give an answer to the issue:

how do I get the token download URL (not the public nor the signed with expiration) with the Admin SDK (from a cloud function for example)

[ghost]

Please urgent help for this issue.

My process steps:

1. Creating a cloud function for creating thumbnails
   https://github.com/firebase/functions-samples/tree/master/generate-thumbnail
   (I have just changed "adding link to database" section from realtime db to firestore)

2. Opening my Flutter app and uploading an image file from phone's gallery

3. Then "creating thumbnails" function executes.
   When I check storage, I see generating thumb is okay and "Download URL"s are:
   image Download URL"
   thumbnail Download URL"

4. That cloud function also adds urls to firestore
   image url
   thumbnail url

5. For a while (7 days?) links on firestore work but at the end of this duration links are expiring...
   The request signature we calculated does not match the signature you provided. Check your Google secret key and signing method.

We are using createThumbnail function for our apps catalog images and that contains too much records.
So we can't delete images (when links expire) and reupload them again weekly.

Is there another solution for getting public links (never expires and can be accessible from my flutter app)?

If I want to get an unexpired link, can I use still signedURL?
If not, how can I add tokenURL system to generateThumnails function?
Otherwise appliying makingPublic() all images and their thumnails for my mobile app is easy to use?

It is demoralising that I would be test if links expire or not in every situation after 7 days.
So I am unable to add new catalog datas and images to my app in this period...

I think there must be few solutions, cause we shouln't be only users who are facing this problem.
So many apps are also using thumbnail creating like us.

Thanks for your cooperation.

[tdkehoe]

I wrote a complete answer to this problem on Stack Overflow: https://stackoverflow.com/questions/42956250/get-download-url-from-file-uploaded-with-cloud-functions-for-firebase/56010225#56010225 I’ve heard of one other way to fix the problem, involving reverting to an older version of the software. I haven’t had time to check that out.

…

-- Thomas David Kehoe LanguageTwo.com Rewire your brain to learn languages by ear From: Kaiser LEKA <notifications@github.com> Reply-To: googleapis/nodejs-storage <reply@reply.github.com> Date: Sunday, September 8, 2019 at 5:53 PM To: googleapis/nodejs-storage <nodejs-storage@noreply.github.com> Cc: Thomas David Kehoe <kehoe@casafuturatech.com>, Comment <comment@noreply.github.com> Subject: Re: [googleapis/nodejs-storage] Get long lived download URLs in addition to SignedURLs (#697) Please urgent help for this issue. My process steps: Creating a cloud function for creating thumbnails https://github.com/firebase/functions-samples/tree/master/generate-thumbnail (I have just changed "adding link to database" section from realtime db to firestore) Opening my Flutter app and uploading an image file from phone's gallery Then "creating thumbnails" function executes. When I check storage, I see generating thumb is okay and "Download URL"s are: image Download URL" thumbnail Download URL" That cloud function also adds urls to firestore image url thumbnail url For a while (7 days?) links on firestore work but at the end of this duration links are expiring... The request signature we calculated does not match the signature you provided. Check your Google secret key and signing method. We are using createThumbnail function for our apps catalog images and that contains too much records. So we can't delete images (when links expire) and reupload them again weekly. Is there another solution for getting public links (never expires and can be accessible from my flutter app)? If I want to get an unexpired link, can I use still signedURL? If not, how can I add tokenURL system to generateThumnails function? Otherwise appliying makingPublic() all images and their thumnails for my mobile app is easy to use? It is demoralising that I would be test if links expire or not in every situation after 7 days. So I am unable to add new catalog datas and images to my app in this period... I think there must be few solutions, cause we shouln't be only users who are facing this problem. So many apps are also using thumbnail creating like us. Thanks for your cooperation. — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

[giammin]

@kaiserleka @tdkehoe @theochampion the only workaround I found is to let the client generate the token download URL

the cloud function write the bucket path and the client the first time it consume that resource will write the download url

it is not an optimal solution but it is the only one available officially (I dont think that using the V2 auth will work forever...)

[ghost]

I am currently trying to get image link on my app like this:
(So I dont use anymore "adding to database section" on generateThumbnail cloud function )

I have added this dart code, (instead of getting data link from firestore)

 
      var _url=await FirebaseStorage.instance.ref()
      .child("catalogItems")
      .child("thumb_${curCatalogItem.id}_${imageData['suffix']}.jpg").getDownloadURL();
      _imageUrl=_url;
    

but if a user is not logged in, it gives no permission to get this url.

When thinking that a list having too many list items (i.e: 100),
I must execute this codes for every catalog item.

Does this way affects performans or anything?

If my way is acceptable, I need to change this:

rules_version = '2';
service firebase.storage {
  match /b/{bucket}/o {
    match /{allPaths=**} {
      allow read, write: if request.auth != null;
    }
  }
}

[ghost]

I have changed storage rules to this (for not loggined users on my app):

rules_version = '2';
service firebase.storage {
  match /b/{bucket}/o {
    match /{allPaths=**} {
      allow read, write: if request.auth != null;
      allow read: if request.auth == null;
    }
  }
}

It is working (I can display images and their thumbails) eventhough it gives this:
error getting token java.util.concurrent.ExecutionException: com.google.firebase.internal.api.FirebaseNoSignedInUserException: Please sign in before trying to get a token.

[giammin]

@kaiserleka you should ask on stackoverflow or other q&a sites. this is not the right place.

that is not related to the issue

or if you think you are experiencing a bug than open a new issue

[ItsaMeTuni]

Why is this closed? There is no solution to the problem and no reason as to why this functionality doesn't exist.
This looks like a quite important feature to many people (including me) and using firebaseStorageDownloadTokens as @tdkehoe said doesn't seem like a very good idea since it's not an official feature and it could disappear or stop working anytime without notice. For now, I'll be using it, but I'd really appreciate it if the team took some time to either support firebaseStorageDownloadTokens officially (documentation, etc) so we can sleep at night knowing our apps won't break out of nowhere or implement a function for getting a permanent URL (which doesn't seem to be very hard to do since the functionality is already there for frontend libraries).

[masonlouchart]

Please, re-open this issue.
We need permanent download tokens or an explanation why we won't get them.
Thank you.

[bartekpacia]

This issue definitely needs to be reopened and suggestion to be implemented. I can't understand why such a simple activity requires so much hassle.

[masonlouchart]

I finally achieved to implement this solution. Be careful to have 2 levels of "metadata". It's not perfect due to the hard coding of the firebase storage URL but it works. Moreover, I think the firebaseStorageDownloadTokens property is not documented.

let options = {
    metadata: {
        metadata: {
            firebaseStorageDownloadTokens: uuid
        }
    }
};
bucket.upload(localFile, options);

[JorgeLTE]

I'm in the same boat. Why is the issue closed?
Currently I'm using V2 signing with long lasting expiration dates, but for how long will V2 be supported?

[bjcooper]

Same boat here. It seems very strange that there is no equivalent of getDownloadUrl() from the admin side. Any explanation would be helpful.

[benjaminbalazs]

Same here. My storage rules file has an explicit metadata check to decide whether the requesting user is amongst the users who can read the file. In other words, I really need the auth object populated in my storage rules file. The only way to achieve this is by using getDownloadUrl(), but I am not on the web.

[alanrubin]

Got to this same issue this morning - needs a way to get persistent urls for downloading files from the storage in the admin side, please reopen this issue.

[todorone]

@AVaksman @fhinkel @bcoe This issue is a real pain for developers. Please consider reopening it or point out to method of getting persistent download urls on server side. Thanks.

[febg11]

end-user shares it with others.

Thanks for chiming in, looking to dig into this properly.

If the getDown

Hi @Danebrouwer97,

You do not believe our requested feature should be implemented because you believe the solution already exists?

That wasn't my intention no. I may have the wrong interpretation as well.

I should take a step back, what originally lead you here with long lived Signed URLs?

My concern with this approach is if a Signed URL is shared with the user and only one is created then it's effectively "public" if an end-user shares it with others.

Thanks for chiming in, looking to dig into this properly.

If the client Side getDownloadURL() is effectively providing a public link even if the file is locked behind storage rules, isn't this the same problem....A user could just generate a link with getDownloadUrl and share the link with other people.

Isn't up to the developer to determine whether this is the behaviour he wants. A function on the node storage package that gets a public url similar to the firebase client getDownloadUrl would be useful.

[frankyn]

Hi @febg11, thanks for replying to this thread.

This library is scoped to support the Storage API and getDownloadUrl() is specific to Storage for Firebase API. Therefore we won't support getDownloadUrl() in this library.

If you're using Storage for Firebase and want to use this library to do the same thing as getDownloadUrl() follow this example (ONLY for Storage for Firebase API and does not work with Storage API):

const bucketName = 'anima-frank';
const filename = 'test.js';

// Imports the Google Cloud client library
const {Storage} = require('@google-cloud/storage');
const UUID = require("uuid-v4");

let uuid = UUID();

// Creates a client
const storage = new Storage();

async function uploadFile() {
  // Uploads a local file to the bucket
  await storage.bucket(bucketName).upload(filename, {
    metadata: {
          contentType: 'text/plain',
          metadata: {
            firebaseStorageDownloadTokens: uuid
          }
    }
  });
  console.log(`${filename} uploaded to ${bucketName}.`);
}

uploadFile().catch(console.error);


// Object is located at:
const fileUrl = `https://firebasestorage.googleapis.com/v0/b/${bucketName}/o/${filename}?alt=media&token=${uuid}`;

console.log(fileUrl);

[giammin]

using a guid for security... this is really wrong

[frankyn]

Hi @giammin, thanks for chiming in.

The Storage for Firebase solution with a GUID is what's used by the service, there are other solutions documented #697 (comment)

[giammin]

sorry @frankyn but it seems that one cant have a proper way to protect a file without creating a custom service

you have public accessible file.
At max one can use "Unguessable" url which is security by obscurity done with guid which are not so "Unguessable"

and anyway the only solution provided to OP is using an undocumented feature

[febg11]

sorry @frankyn but it seems that one cant have a proper way to protect a file without creating a custom service

you have public accessible file.
At max one can use "Unguessable" url which is security by obscurity done with guid which are not so "Unguessable"

and anyway the only solution provided to OP is using an undocumented feature

I agree but Doug Stevenson said this here.

"It's not supported case to build a URL like this. It may work today, but could break in the future. You should only use the provided APIs to generate a proper download URL"

It seems that all the solutions are hacky/unsafe... is there not a better solution?

[frankyn]

Hi @giammin and @febg11,

Thanks for the feedback. Could you both restate which the you're trying to solve? I want to make sure I'm not missing anything.

[febg11]

Hi @frankyn.

I am just looking for a solution that allows me to generate a url that will point to a file on firebase storage that will not expire and that can be generated in node.js. I will store this url in a cloud firestore document so users can read it instead than pointlessly calling getDowloadURL() over and over and wasting resources.

I would rather call a method that is defined in the cloud storage node library rather than implementing my own workaround they may not be secure/may break at any time.

Being able to revoke the link is not a problem for me, however I cannot speak for @giammin requirements.

Thanks for prompt reply.

[frankyn]

Thanks @febg11, I chimed in on a related question here: firebase/firebase-js-sdk#76 to address this gap. I can understand not wanting to rely on something that may change in the future for unforeseen circumstances.

Pending response from @giammin.

[giammin]

Hi @frankyn thanks for keeping the interest In this issue.

My need is to create image and file from a cloud function and from an external service (netcore)

These files are consumed by an android/iOS app (nativescript with firebase as backend)

Users upload files in the Firebase storage (profile pic or attachment) and those files need to be checked and elaborated and then are available to some other users.

But the service or cloud functions can’t create a protected link to those files. I need the client app to generate those public urls.

Another problem is that those files should not be public accessible but only to selected users. It is not acceptable that anyone with the right link can download them And the storage security rules does not apply to download link

To get around this problem I create a netcore service that works as a reverse proxy for the firebase storage.
It provide public link for the files and authorization is handled with tokens that are send with the request. Why can’t firebase storage work this way? It also have the user token and for authorization one can take advantage of the security rules...

I find those downloadurls cumbersome. If one is leaked or need to change you have to update every reference to it

[frankyn]

Thanks @giammin, spoke with folks on the Storage for Firebase team last week and they're looking into addressing protected access in firebase/firebase-js-sdk#76 (which I think addresses your issue) and considered a feature gap in their support.

Could you raise this feedback in that issue in case there's a path forward that I'm not aware of in that firebase issue? @avolkovi, can probably add more here as well.

[iingles]

I think I'm having a similar issue to what many people here are having.

I'm trying to implement a "search" function for people to search through proprietary PDFs stored in Firebase Storage. I don't want to allow anyone to be able to download the file unless they are authenticated. So far I haven't been able to find a way to do this without extensive workarounds - I don't fully trust getDownloadURL() or the signed URL because all someone has to do is share the URL and it's public (even if you set the life to the signed URL to be very short). This seems like a gaping hole in Firebase security to me. I'll explore some of the workarounds mentioned here but to be honest I need to get this project out and done and I might abandon Firebase altogether if I can't keep the files completely secure.

[nicoqh]

@iingles

I don't fully trust getDownloadURL() or the signed URL because all someone has to do is share the URL and it's public (even if you set the life to the signed URL to be very short).

Be careful, even a signed URL can expose the file forever due to how Cloud Storage for Firebase uses download tokens.

More info: https://www.sentinelstand.com/article/guide-to-firebase-storage-download-urls-tokens

[avolkovi]

@nicoqh GCS Signed URLs are completely distinct from Cloud Storage for Firebase Download URLs. They go through different APIs (one is a Cloud API intended for backend based authentication or Google OAuth, the other is a Firebase API intended for client based authentication such as Firebase Authentication)

The idea behind Firebase Download URLs is that if someone has access to the link, it's true that they could share the link and thereby share access, but they could just as easily download the file and share it through some other means. Having access to the download URL is equivalent from a security standpoint as having permanent access to the file. If you upload a new version of the file, we regenerate the download URL, thereby preventing anyone with access to the previous version from also accessing the new version.

Hope that helps clarify the distinction a bit.

[nicoqh]

@avolkovi My point was that a signed URL will expose your Firebase download token because Cloud Storage for Firebase uses Cloud Storage's concept of metadata to store its download tokens, and Cloud Storage exposes metadata through its response headers. So a user will be able to construct a long-lived Firebase download URL (https://firebasestorage.googleapis.com/v0/b/...) from a short-lived signed Google Storage URL (https://storage.googleapis.com/...) even if they don't have read permissions (getDownloadUrl()) per the Firebase Storage security rules.

It's something to be aware of since signed URLs are suggested as an alternative to long-lived download URLs.

[avolkovi]

@nicoqh Good point! That is correct.

One thing to note, if the Firebase API is not being used, the download tokens will not be generated. They are only generated when uploading via the Firebase API or when calling getDownloadUrl() or getMetadata() via the Firebase SDK. If you're just using GCS signed URLs without Firebase in the mix then you won't have this issue. Yet another reason for us to rethink storing these in Metadata....

[frankyn]

Closing out issue for now, will reopen if there's another option here.

[axelvaindal]

Hello @frankyn,
Sorry if this is not the right place to ask, I've been following this thread for quite a long time and I've encountered the same issues than other developers when it comes to getSignedUrl for long-lived URLs.

Context

We are trying to build some UI on the web, around video files uploaded by our customers through our android and ios apps.
We upload the files using the Storage API from Firebase and once the upload is complete, a cloud function runs automatically to store our file in our database along with the metadata.
We originally tried to create signedUrl but we realize they stopped working after some time due to the expiration of the default service account, as you mentioned in the thread.
We then used your suggestion to use the mediaLink from the getMetadata function in our cloud function through the Storage API, and it's working almost fine. We have now a public file that is (up to now) correctly interpreted by the browser <video> tag when we fetch the data client-side.

Problem

However, when this same link is shared through a social network (for example) Twitter, as part of an og:video meta tag or twitter:video, we get a weird error:

Anonymous caller does not have storage.objects.get access to the Google Cloud Storage object.

I don't fully understand why Google Chrome, Safari, and other browsers are able to read this media link properly and anyone can download the full video when using the mediaLink, but the preview on social network is broken like that.
It was not the case with getSignedUrl but unfortunately, those stops working after ~2 weeks due to renewal of the service account used to generate them (default).

Do you have any insight or can you show me the right way to handle such use cases ?

I really appreciate any help you can provide.

[nahueld-owners]

it's incredible that this is not an out of the box feature specially coming from Google, either I'm too stupid to want long lived tokens for resources or this is a hole in the documentation / missing feature

[nlptechguy]

Based on the comments I've read, it appears that people are enthusiastic about having this feature, while Google seems reluctant to provide it. It's as straightforward as that.

[MorenoMdz]

Bumping this one, there is no reason to not implement this.

[KemikalGeneral]

Just in case anyone - like me - has spent hours trying to work out why I can upload an image client-side and get a token, but cannot if uploaded server-side (node), and feels like stabbing themselves in the face with a rusty nail, this is what I've done:

• Upgrade to Admin SDK v11.10+

import {getDownloadURL} from "firebase-admin/storage";

const IMAGE_FILE_PATH = `profilePictures/${path.basename(PROCESSED_IMAGE_PATH)}`
const [uploadedFile] = await bucket.upload(PROCESSED_IMAGE_PATH, { destination: IMAGE_FILE_PATH })
const LINK_WITH_TOKEN = await getDownloadURL(uploadedFile);

Which gives me the download link with the media token:

https://firebasestorage.googleapis.com/v0/b/my-app.appspot.com/o/profilePictures%imageName.webp?alt=media&token=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx

[rscotten]

Just in case anyone - like me - has spent hours trying to work out why I can upload an image client-side and get a token, but cannot if uploaded server-side (node), and feels like stabbing themselves in the face with a rusty nail, this is what I've done:

• Upgrade to Admin SDK v11.10+

import {getDownloadURL} from "firebase-admin/storage";

const IMAGE_FILE_PATH = `profilePictures/${path.basename(PROCESSED_IMAGE_PATH)}`
const [uploadedFile] = await bucket.upload(PROCESSED_IMAGE_PATH, { destination: IMAGE_FILE_PATH })
const LINK_WITH_TOKEN = await getDownloadURL(uploadedFile);

Which gives me the download link with the media token:

https://firebasestorage.googleapis.com/v0/b/my-app.appspot.com/o/profilePictures%imageName.webp?alt=media&token=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx

THANK YOU!! Wow, I almost didn't read to the very end of this thread.

[KemikalGeneral]

You're welcome @rscotten 😁