invalid character '\x1f' looking for beginning of value when uploading via java apis #142
Comments
|
Just an update on this, as a workaround I decided to try to create a bucket with a file when starting up the container, as documented in the read me, after which I tried doing I'm now not able to create files in it due to #84 but that's a different issue. |
|
Hey @DanieleSassoli, sorry for the delayed response. Yeah currently we don't support the GRPC API. I haven't looked into how much work that would be, do you have an idea? If not, I can take a look over the holidays. |
|
Opened #146 to track support for the grpc api |
|
Hey Team. I have been investigating this issue. I am leaning towards a gzip compression issue. I am digging through the Java code and it clearly looks like HTTP. Furthermore, the \x1f is likely a character you would see in a gzipped POST payload over HTTP. See screenshot attached.
|
|
@fsouza , a follow-up. I simply hacked the debugger and set |
|
@dnatic09 I've made a bit of progress based on your finding. I was able to patch the public class StorageEmulatorRpc extends HttpStorageRpc {
public StorageEmulatorRpc(StorageOptions options) {
super(options);
try {
var field = FieldUtils.getField(super.getClass(), "storage", true);
field.setAccessible(true);
var transportOptions = (HttpTransportOptions) options.getTransportOptions();
var transport = transportOptions.getHttpTransportFactory().create();
var initializer = transportOptions.getHttpRequestInitializer(options);
var storage = new Storage.Builder(transport, new JacksonFactory(), initializer)
.setRootUrl(options.getHost())
.setGoogleClientRequestInitializer(request -> request.setDisableGZipContent(true))
.setApplicationName(options.getApplicationName())
.build();
field.set(this, storage);
} catch (Exception e) {
throw new RuntimeException(e);
}
}
}I then specified a custom var builder = StorageOptions.getDefaultInstance().toBuilder();
builder.setHost(clientOptions.host());
builder.setServiceRpcFactory(StorageEmulatorRpc::new);
builder.setTransportOptions(HttpTransportOptions.newBuilder()
.setHttpTransportFactory(new InsecureHttpTransportFactory())
.build());
return builder.build().getService();This allowed me to successfully create both buckets and blobs using the generated storage client: Obviously it would be superior to just handle gzip on the server side, but I'm going to continue testing with this patch tomorrow to see if I can get this emulator working in a java environment. |
|
@flylo , great work here! I think we may get some traction. Getting Google to accept a PR will likely be difficult. On the flip side, there are only a handful of methods that need to be changed on the server to check for the I am not familiar in GO, but I imagine it will not be difficult to change. If I have spare time, I'll try it out and submit a PR against this repo. |
|
@dnatic09 Sorry for the confusion... when I said "server-side", I meant the server inside of this GCS emulator. The problem I'm running into now is this... The response we get from the emulator for creating a Which results in a Specifically, it fails the return new BlobId(checkNotNull(bucket), checkNotNull(name), generation);I assume the fix here is simple, but I've also never used GO and will need some time to read the code. @fsouza any ideas? |
|
@flylo , I ran into the same problem. The funny thing is that you get an NPE, but the file is actually in the fake GCS server and can be pulled. I already tested it. We should definitely help fix it, but there is definitely a path forward here. |
|
Have made it a little further, but now we're running into issues where the golang server is returning uppercased field names that the java JSON parser doesn't catch: I'm looking for a way to set global JSON-serialization settings but haven't found it yet. |
|
@flylo , GOT IT ! Follow this: fake-gcs-server/fakestorage/response.go Lines 35 to 39 in 297b8cd Then, you can change how the field is serialized. We do this and the casing will align! |
|
@dnatic09 nice! Yeah if there's no way to specify a global serialization option (like how you'd inject an |
|
@flylo , I think Java users around the world would be WAY better off with just create-bucket and put-file functions working. The rest can be tacked on later. Your help is MUCH appreciated. There is no way I could have gotten to this anytime soon. I am so backed up with work. |
|
hey @flylo @dnatic09 thanks for looking into this and apologies for being absent. So a quick of notes / questions:
(as a side note, there isn't a way to change that behavior globally, it's one of the issues with Go's struct tags and why I am calling it a mistake to use a third-party struct in our JSON response - our struct is not exported, so we can change this without introducing a breaking change though) |
|
@fsouza I'm not sure why that was happening if the google go library defines the struct tags... Maybe it has something to do with the fact that you're serializing an array of those |
|
@fsouza , can you trigger a DockerHub push? (Push image 1.22.0 as a docker tag in DockerHub) EDIT: I think I can use |
|
@dnatic09 ohh it should be there, it's supposed to be automatic. I'll check what's up with github actions. |
|
@fsouza , no joy. I purged |
|
@dnatic09 do you have a small reproducer that I can run locally? This way I can figure something out that will definitely work (+ maybe add it as a runnable sample in CI). I'm afk right now, but I'll try to fix this later today once I get to the computer (and figure out what's up with the automatic tagging) Thanks for checking back! |
|
I can make a reproducer overnight. I will message back when it is ready. |
|
https://github.com/dnatic09/Fake-Gcs-Test Just clone and run One test will pass (rigged CURL). |
|
Hey team. Just checking in. Any updates? |
|
Hey @dnatic09, sorry ended-up taking an extended break but I'm back this week. I'll look at your example later today or tomorrow and see if I can figure out what's up. |
|
No worries. Today is my first day back into work since Christmas. Completely understand. Thank you again for engaging on this. |
|
Ohhh I see what's the problem here. I enabled compressed responses, but compressed requests are not standard and we have to implement it manually. Ugh. |
|
@dnatic09 tests are passing now 😁 I was able to include a test for this, so will skip adding a Java example. Closing this issue for now, feel free to reopen or to open a new one if something doesn't work for you. |
Hi, I'm trying to create a bucket in the fake gcs server using the java apis(which don't seem to be documented, happy to contribute with some docs once I get this working) but getting
This is the code:
Can't really figure out what I'm doing wrong, does anybody have any idea?
The text was updated successfully, but these errors were encountered: